[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:37:42 dist-test-slave-m5z1 krb5kdc[26568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:37:42 dist-test-slave-m5z1 krb5kdc[26568](info): set up 2 sockets
Sep 24 13:37:42 dist-test-slave-m5z1 krb5kdc[26568](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 13:37:43.885636 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:37:43 dist-test-slave-m5z1 krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:37:44.002525 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35013
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:35013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:44.081310 26581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:44.081524 26581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:44.081560 26581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:44.082705 26581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:37:44.082774 26581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:44.082801 26581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:44.082834 26581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:37:44.082854 26581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:37:44.084283 26581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:35013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:44.084539 26581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:44.084760 26581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:44.089617 26587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:44.089649 26586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:44.089653 26581 server_base.cc:1034] running on GCE node
W20260924 13:37:44.089635 26589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:44.090097 26581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:44.090399 26581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:44.091566 26581 hybrid_clock.cc:648] HybridClock initialized: now 1790257064091547 us; error 36 us; skew 500 ppm
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:44.094012 26581 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:37:44.094400 26581 webserver.cc:460] Webserver started at http://127.25.240.126:36953/ using document root <none> and password file <none>
I20260924 13:37:44.094589 26581 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:44.094627 26581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:44.094715 26581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:44.095685 26581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "f9c9e3769e474358a79c606f8aba0b2c"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "00281a3589910dd79583efc97ef189be"
server_key_iv: "9ddfc242ca69b85250bad8eafc27623f"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.095973 26581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "f9c9e3769e474358a79c606f8aba0b2c"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "00281a3589910dd79583efc97ef189be"
server_key_iv: "9ddfc242ca69b85250bad8eafc27623f"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.097447 26581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:37:44.098232 26596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.098416 26581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:44.098495 26581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "f9c9e3769e474358a79c606f8aba0b2c"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "00281a3589910dd79583efc97ef189be"
server_key_iv: "9ddfc242ca69b85250bad8eafc27623f"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.098551 26581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:44.121779 26581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:44.124392 26581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:44.124501 26581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:44.128508 26581 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:35013
I20260924 13:37:44.128567 26648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:35013 every 8 connection(s)
I20260924 13:37:44.128917 26581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:37:44.129844 26649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:44.132040 26649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c: Bootstrap starting.
I20260924 13:37:44.132615 26649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:44.132900 26649 log.cc:824] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:44.133667 26649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c: No bootstrap required, opened a new log
I20260924 13:37:44.135200 26649 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c9e3769e474358a79c606f8aba0b2c" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35013 } }
I20260924 13:37:44.135378 26649 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:44.135455 26649 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c9e3769e474358a79c606f8aba0b2c, State: Initialized, Role: FOLLOWER
I20260924 13:37:44.135617 26649 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c9e3769e474358a79c606f8aba0b2c" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35013 } }
I20260924 13:37:44.135697 26649 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:37:44.135739 26649 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:37:44.135788 26649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:44.136286 26649 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c9e3769e474358a79c606f8aba0b2c" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35013 } }
I20260924 13:37:44.136409 26649 leader_election.cc:302] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9c9e3769e474358a79c606f8aba0b2c; no voters:
I20260924 13:37:44.136588 26649 leader_election.cc:288] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:37:44.136626 26652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:44.136797 26652 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [term 1 LEADER]: Becoming Leader. State: Replica: f9c9e3769e474358a79c606f8aba0b2c, State: Running, Role: LEADER
I20260924 13:37:44.136834 26649 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:37:44.136899 26652 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c9e3769e474358a79c606f8aba0b2c" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35013 } }
I20260924 13:37:44.137568 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 26581
I20260924 13:37:44.137677 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:37:44.138314 26561 external_mini_cluster.cc:1428] Setting key 2a02301fa3bb27fdbfa9c5e354dba394
W20260924 13:37:44.138744 26667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:37:44.138780 26667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:37:44.139103 26653 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9c9e3769e474358a79c606f8aba0b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c9e3769e474358a79c606f8aba0b2c" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35013 } } }
I20260924 13:37:44.139333 26653 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [sys.catalog]: This master's current role is: LEADER
I20260924 13:37:44.139273 26654 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9c9e3769e474358a79c606f8aba0b2c. Latest consensus state: current_term: 1 leader_uuid: "f9c9e3769e474358a79c606f8aba0b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c9e3769e474358a79c606f8aba0b2c" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35013 } } }
I20260924 13:37:44.139536 26654 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c [sys.catalog]: This master's current role is: LEADER
I20260924 13:37:44.139642 26671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:37:44.139819 26671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:37:44.141366 26671 catalog_manager.cc:1347] Generated new cluster ID: 03a40ac5c40e419e8b84f9a3eff357d0
I20260924 13:37:44.141417 26671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:44.153010 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.139189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.25.240.126:35013)
0924 13:37:44.139436 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:44.139445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:44.139754 (+ 309us) server_negotiation.cc:407] Connection header received
0924 13:37:44.141999 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:44.142005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:44.142054 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:44.142186 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:44.143564 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.144487 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.145519 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.146032 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.146171 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.149453 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:44.149457 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:44.149457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:44.149467 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:44.151134 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.151584 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.151587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.151587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.151675 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.151936 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.151937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.151937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.152174 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:44.152326 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:44.152618 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 13:37:44.152746 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":70,"threads_started":1}
I20260924 13:37:44.157087 26671 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:37:44.157661 26671 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:37:44.166412 26671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9c9e3769e474358a79c606f8aba0b2c: Generated new TSK 0
I20260924 13:37:44.166599 26671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:37:44.190809 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35013
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:44.270502 26675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:44.270735 26675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:44.270773 26675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:44.271909 26675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:44.271974 26675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:44.272020 26675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:37:44.273592 26675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35013
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:44.273850 26675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:44.274088 26675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:37:44.278916 26675 server_base.cc:1034] running on GCE node
W20260924 13:37:44.278908 26683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:44.278899 26681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:44.278887 26680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:44.279317 26675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:44.279574 26675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:44.280712 26675 hybrid_clock.cc:648] HybridClock initialized: now 1790257064280693 us; error 34 us; skew 500 ppm
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:44.283156 26675 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:37:44.283571 26675 webserver.cc:460] Webserver started at http://127.25.240.65:45483/ using document root <none> and password file <none>
I20260924 13:37:44.283802 26675 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:44.283847 26675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:44.283977 26675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:44.284982 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "71a91a91822347ad97ac2fe7f2768299"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "1b1af09bf4d5f700d534ebd5fedf5c37"
server_key_iv: "e83acd9adeb3b13a4f7da985cf89c44f"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.285327 26675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "71a91a91822347ad97ac2fe7f2768299"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "1b1af09bf4d5f700d534ebd5fedf5c37"
server_key_iv: "e83acd9adeb3b13a4f7da985cf89c44f"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.286788 26675 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:37:44.287554 26690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.287757 26675 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:44.287842 26675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "71a91a91822347ad97ac2fe7f2768299"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "1b1af09bf4d5f700d534ebd5fedf5c37"
server_key_iv: "e83acd9adeb3b13a4f7da985cf89c44f"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.287899 26675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:44.305033 26675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:44.307111 26675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:44.307242 26675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:44.307425 26675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:37:44.307735 26675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:44.307768 26675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.307789 26675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:44.307840 26675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.313469 26675 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:41285
I20260924 13:37:44.313544 26803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:41285 every 8 connection(s)
I20260924 13:37:44.313894 26675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:37:44.315626 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 26675
I20260924 13:37:44.315739 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:37:44.315918 26561 external_mini_cluster.cc:1428] Setting key 3130dab1deffdd2aff1ec1ffd4f5761d
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
I20260924 13:37:44.323757 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.314592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:44625 (local address 127.25.240.126:35013)
0924 13:37:44.314633 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:44.314635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:44.314817 (+ 182us) server_negotiation.cc:407] Connection header received
0924 13:37:44.315328 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:44.315329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:44.315342 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:44.315399 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:44.316585 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.317090 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.318536 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.318746 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.318806 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.320900 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:44.320901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:44.320901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:44.320910 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:44.322396 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.322795 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.322796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.322796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.322860 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.323110 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.323110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.323111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.323223 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:44.323298 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:44.323574 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 13:37:44.323668 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:37:44.323967 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.314557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35013 (local address 127.25.240.65:44625)
0924 13:37:44.314689 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:44.314733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:44.315226 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:44.315474 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:44.315479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:44.316226 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:44.316503 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:44.316510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.317187 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.317188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:44.318441 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:44.318444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.318828 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.318829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:44.318956 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.319476 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:44.319486 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:44.320794 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:44.322488 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:44.322495 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:44.322497 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:44.322719 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:44.322930 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:44.322931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:44.322931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:44.323038 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:44.323290 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:44.323297 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:44.323439 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 13:37:44.323538 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 13:37:44.323692 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
I20260924 13:37:44.324422 26804 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35013
I20260924 13:37:44.324513 26804 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:44.324683 26804 heartbeater.cc:511] Master 127.25.240.126:35013 requested a full tablet report, sending...
I20260924 13:37:44.325242 26613 ts_manager.cc:194] Registered new tserver with Master: 71a91a91822347ad97ac2fe7f2768299 (127.25.240.65:41285)
I20260924 13:37:44.325953 26613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:44625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:37:44.340431 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35013
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:44.420753 26809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:44.420940 26809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:44.420964 26809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:44.422022 26809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:44.422056 26809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:44.422081 26809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:37:44.423382 26809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35013
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:44.423549 26809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:44.423709 26809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:37:44.428640 26809 server_base.cc:1034] running on GCE node
W20260924 13:37:44.428601 26817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:44.428742 26815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:44.428661 26814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:44.429098 26809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:44.429335 26809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:44.430490 26809 hybrid_clock.cc:648] HybridClock initialized: now 1790257064430477 us; error 28 us; skew 500 ppm
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:44.432847 26809 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:37:44.433251 26809 webserver.cc:460] Webserver started at http://127.25.240.66:41789/ using document root <none> and password file <none>
I20260924 13:37:44.433470 26809 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:44.433537 26809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:44.433684 26809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:44.434686 26809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "42673e92d3b8452abc94513f4446b5c2"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "026b4a4adb633ab388c50fcfc94b73b9"
server_key_iv: "f22fb61a6752822aadd536c402101fc9"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.435041 26809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "42673e92d3b8452abc94513f4446b5c2"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "026b4a4adb633ab388c50fcfc94b73b9"
server_key_iv: "f22fb61a6752822aadd536c402101fc9"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.436424 26809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:37:44.437140 26824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.437330 26809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:44.437428 26809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "42673e92d3b8452abc94513f4446b5c2"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "026b4a4adb633ab388c50fcfc94b73b9"
server_key_iv: "f22fb61a6752822aadd536c402101fc9"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.437520 26809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:44.453241 26809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:44.455407 26809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:44.455578 26809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:44.455817 26809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:37:44.456146 26809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:44.456200 26809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.456240 26809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:44.456269 26809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.461932 26809 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:39757
I20260924 13:37:44.461971 26937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:39757 every 8 connection(s)
I20260924 13:37:44.462329 26809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:37:44.464926 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 26809
I20260924 13:37:44.464998 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:37:44.465171 26561 external_mini_cluster.cc:1428] Setting key 28416060f1491099a2ef25e5e3615993
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:44.472899 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.463022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:50091 (local address 127.25.240.126:35013)
0924 13:37:44.463077 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:44.463079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:44.463337 (+ 258us) server_negotiation.cc:407] Connection header received
0924 13:37:44.463907 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:44.463909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:44.463922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:44.463969 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:44.464921 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.465555 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.467091 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.467347 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.467431 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.469654 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:44.469656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:44.469656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:44.469665 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:44.471353 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.471739 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.471741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.471741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.471811 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.472086 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.472087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.472088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.472259 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:44.472336 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:44.472678 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 13:37:44.472771 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 13:37:44.472924 26940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.463011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35013 (local address 127.25.240.66:50091)
0924 13:37:44.463189 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:44.463225 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:44.463790 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:44.464049 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:44.464059 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:44.464614 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:44.464802 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:44.464807 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.465659 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.465660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:44.467005 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:44.467008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.467453 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.467454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:44.467575 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.468120 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:44.468128 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:44.469524 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:44.471437 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:44.471443 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:44.471445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:44.471663 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:44.471885 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:44.471886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:44.471886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:44.472006 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:44.472397 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:44.472402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:44.472530 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 13:37:44.472627 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 13:37:44.472751 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260924 13:37:44.473475 26938 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35013
I20260924 13:37:44.473611 26938 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:44.473778 26938 heartbeater.cc:511] Master 127.25.240.126:35013 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
I20260924 13:37:44.474165 26613 ts_manager.cc:194] Registered new tserver with Master: 42673e92d3b8452abc94513f4446b5c2 (127.25.240.66:39757)
I20260924 13:37:44.474570 26613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:50091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:37:44.492615 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35013
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:44.572106 26943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:44.572393 26943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:44.572420 26943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:44.574002 26943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:44.574054 26943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:44.574113 26943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:37:44.575814 26943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35013
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:44.576083 26943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:44.576327 26943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:44.581553 26948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:44.581580 26951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:44.581593 26949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:44.581764 26943 server_base.cc:1034] running on GCE node
I20260924 13:37:44.582024 26943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:44.582221 26943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:44.583364 26943 hybrid_clock.cc:648] HybridClock initialized: now 1790257064583336 us; error 50 us; skew 500 ppm
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:44.585922 26943 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:37:44.586387 26943 webserver.cc:460] Webserver started at http://127.25.240.67:42651/ using document root <none> and password file <none>
I20260924 13:37:44.586633 26943 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:44.586709 26943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:44.586871 26943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:44.587926 26943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "cc4b6bbc8ef64724ac8d64fcbfda7aef"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "23f3eb6cc27455ae425c42ba3c89852e"
server_key_iv: "099f1044001314f482f3c1b0fe7f1bb1"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.588303 26943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "cc4b6bbc8ef64724ac8d64fcbfda7aef"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "23f3eb6cc27455ae425c42ba3c89852e"
server_key_iv: "099f1044001314f482f3c1b0fe7f1bb1"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.589670 26943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:37:44.590456 26958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.590663 26943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:37:44.590763 26943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "cc4b6bbc8ef64724ac8d64fcbfda7aef"
format_stamp: "Formatted at 2026-09-24 13:37:44 on dist-test-slave-m5z1"
server_key: "23f3eb6cc27455ae425c42ba3c89852e"
server_key_iv: "099f1044001314f482f3c1b0fe7f1bb1"
server_key_version: "encryptionkey@0"
I20260924 13:37:44.590852 26943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:44.610735 26943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:44.612525 26943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:44.612682 26943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:44.612903 26943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:37:44.613194 26943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:44.613248 26943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.613301 26943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:44.613327 26943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:44.621099 26943 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:33415
I20260924 13:37:44.621132 27071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:33415 every 8 connection(s)
I20260924 13:37:44.621510 26943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:37:44.627314 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 26943
I20260924 13:37:44.627424 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:44.627614 26561 external_mini_cluster.cc:1428] Setting key 09d9c146e85e7f846876689016a3af04
I20260924 13:37:44.630792 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.622270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:34675 (local address 127.25.240.126:35013)
0924 13:37:44.622342 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:44.622344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:44.622543 (+ 199us) server_negotiation.cc:407] Connection header received
0924 13:37:44.622986 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:44.622987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:44.622998 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:44.623065 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:44.623962 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.624508 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.625456 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.625757 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.625819 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.628116 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:44.628118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:44.628118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:44.628124 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:44.629526 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.629979 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.629979 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.629980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.630029 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.630283 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.630283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.630283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.630381 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:44.630430 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:44.630668 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 13:37:44.630727 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 13:37:44.630977 27074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.622237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35013 (local address 127.25.240.67:34675)
0924 13:37:44.622393 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:44.622433 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:44.622894 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:44.623137 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:44.623141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:44.623699 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:44.623882 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:44.623887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.624587 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.624587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:44.625388 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:44.625390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.625832 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.625832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:44.625939 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.626383 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:44.626391 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:44.628004 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:44.629671 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:44.629676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:44.629678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:44.629911 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:44.630089 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:44.630090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:44.630090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:44.630195 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:44.630438 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:44.630442 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:44.630556 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 13:37:44.630643 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 13:37:44.630766 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260924 13:37:44.631444 27072 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35013
I20260924 13:37:44.631531 27072 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:44.631711 27072 heartbeater.cc:511] Master 127.25.240.126:35013 requested a full tablet report, sending...
I20260924 13:37:44.632047 26613 ts_manager.cc:194] Registered new tserver with Master: cc4b6bbc8ef64724ac8d64fcbfda7aef (127.25.240.67:33415)
I20260924 13:37:44.632459 26613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:34675
I20260924 13:37:44.639801 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:37:44.662047 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.654444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.25.240.65:41285)
0924 13:37:44.654599 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:44.654604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:44.654621 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:37:44.654647 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:44.654649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:44.654662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:44.654735 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:44.655212 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.655930 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.656723 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.657061 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.657119 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.659428 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:44.659436 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:44.659436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:44.659443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:44.661037 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.661345 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.661347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.661347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.661400 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.661598 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.661599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.661599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.661730 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:44.661874 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:44.661909 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 13:37:44.661972 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
W20260924 13:37:44.662416 26718 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33484
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[26568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:37:44.691075 27079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:44.676919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.25.240.65:41285)
0924 13:37:44.676980 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:44.676981 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:44.677032 (+ 51us) server_negotiation.cc:407] Connection header received
0924 13:37:44.677084 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:44.677084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:44.677095 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:44.677142 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:44.677712 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.678174 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.678911 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:44.679097 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:44.679146 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:44.687958 (+ 8812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:44.687964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:44.687965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:44.687972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:44.689747 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.690170 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.690171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.690171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.690232 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:44.690474 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:44.690475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:44.690476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:44.690603 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:44.690757 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:44.690879 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 13:37:44.690965 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:37:44.691566 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 26675
I20260924 13:37:44.696944 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 26809
I20260924 13:37:44.703399 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 26943
I20260924 13:37:44.707588 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 26581
2026-09-24T13:37:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1859 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[27086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[27086](info): set up 2 sockets
Sep 24 13:37:44 dist-test-slave-m5z1 krb5kdc[27086](info): commencing operation
krb5kdc: starting...
W20260924 13:37:45.737270 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:37:45 dist-test-slave-m5z1 krb5kdc[27086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:37:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:37:45.852578 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38545
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:38545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:45.931516 27099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:45.931715 27099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:45.931736 27099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:45.932936 27099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:37:45.932979 27099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:45.932993 27099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:45.933007 27099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:37:45.933060 27099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:37:45.934504 27099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:38545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:45.934748 27099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:45.935024 27099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:45.939762 27105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:45.939906 27099 server_base.cc:1034] running on GCE node
W20260924 13:37:45.939858 27107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:45.939762 27104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:45.940219 27099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:45.940508 27099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:45.941617 27099 hybrid_clock.cc:648] HybridClock initialized: now 1790257065941599 us; error 48 us; skew 500 ppm
Sep 24 13:37:45 dist-test-slave-m5z1 krb5kdc[27086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257065, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:45.944008 27099 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:37:45.944469 27099 webserver.cc:460] Webserver started at http://127.25.240.126:36817/ using document root <none> and password file <none>
I20260924 13:37:45.944689 27099 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:45.944753 27099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:45.944916 27099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:45.946019 27099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "5e788b0943b3451e9381dc428ae94eed"
format_stamp: "Formatted at 2026-09-24 13:37:45 on dist-test-slave-m5z1"
server_key: "06d275e2a08477b3824c82ce0ee21338"
server_key_iv: "488a49746cd380aefe035bdf5f8cc7d0"
server_key_version: "encryptionkey@0"
I20260924 13:37:45.946375 27099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "5e788b0943b3451e9381dc428ae94eed"
format_stamp: "Formatted at 2026-09-24 13:37:45 on dist-test-slave-m5z1"
server_key: "06d275e2a08477b3824c82ce0ee21338"
server_key_iv: "488a49746cd380aefe035bdf5f8cc7d0"
server_key_version: "encryptionkey@0"
I20260924 13:37:45.947664 27099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:37:45.948374 27114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:45.948553 27099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:45.948649 27099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "5e788b0943b3451e9381dc428ae94eed"
format_stamp: "Formatted at 2026-09-24 13:37:45 on dist-test-slave-m5z1"
server_key: "06d275e2a08477b3824c82ce0ee21338"
server_key_iv: "488a49746cd380aefe035bdf5f8cc7d0"
server_key_version: "encryptionkey@0"
I20260924 13:37:45.948731 27099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:45.980381 27099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:45.982339 27099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:45.982502 27099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:45.986399 27099 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:38545
I20260924 13:37:45.986441 27166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:38545 every 8 connection(s)
I20260924 13:37:45.986822 27099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:37:45.987425 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 27099
I20260924 13:37:45.987511 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:37:45.987681 26561 external_mini_cluster.cc:1428] Setting key 2cf85fc88aae5d99a866a8e424c83912
I20260924 13:37:45.987838 27167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:45.991190 27167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed: Bootstrap starting.
I20260924 13:37:45.991920 27167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:45.992274 27167 log.cc:824] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:45.993043 27167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed: No bootstrap required, opened a new log
Sep 24 13:37:45 dist-test-slave-m5z1 krb5kdc[27086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:45.995431 27167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e788b0943b3451e9381dc428ae94eed" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } }
I20260924 13:37:45.995618 27167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:45.995663 27167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e788b0943b3451e9381dc428ae94eed, State: Initialized, Role: FOLLOWER
I20260924 13:37:45.995807 27167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e788b0943b3451e9381dc428ae94eed" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } }
I20260924 13:37:45.995887 27167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:37:45.995909 27167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:37:45.995986 27167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:45.996656 27167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e788b0943b3451e9381dc428ae94eed" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } }
I20260924 13:37:45.996747 27167 leader_election.cc:302] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e788b0943b3451e9381dc428ae94eed; no voters:
I20260924 13:37:45.996893 27167 leader_election.cc:288] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:37:45.996937 27172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:45.997058 27172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [term 1 LEADER]: Becoming Leader. State: Replica: 5e788b0943b3451e9381dc428ae94eed, State: Running, Role: LEADER
I20260924 13:37:45.997162 27167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:37:45.997145 27172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e788b0943b3451e9381dc428ae94eed" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } }
I20260924 13:37:45.997488 27172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e788b0943b3451e9381dc428ae94eed. Latest consensus state: current_term: 1 leader_uuid: "5e788b0943b3451e9381dc428ae94eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e788b0943b3451e9381dc428ae94eed" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } } }
I20260924 13:37:45.997601 27172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [sys.catalog]: This master's current role is: LEADER
I20260924 13:37:45.997501 27174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e788b0943b3451e9381dc428ae94eed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e788b0943b3451e9381dc428ae94eed" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } } }
I20260924 13:37:45.997774 27174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed [sys.catalog]: This master's current role is: LEADER
W20260924 13:37:45.998970 27188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:37:45.999166 27188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:37:45.999218 27178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:37:45.999374 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:45.988217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.25.240.126:38545)
0924 13:37:45.988414 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:45.988423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:45.988457 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:37:45.989007 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:45.989014 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:45.989065 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:45.989178 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:45.990010 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:45.990892 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:45.991943 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:45.992464 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:45.992600 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:45.995300 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:45.995303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:45.995304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:45.995313 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:45.997455 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:45.997905 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:45.997907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:45.997908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:45.997994 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:45.998244 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:45.998245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:45.998246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:45.998600 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:45.998775 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:45.998896 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 13:37:45.999051 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260924 13:37:45.999390 27178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:37:46.000975 27178 catalog_manager.cc:1347] Generated new cluster ID: 8a55316043d145f1b082bb61f69efd93
I20260924 13:37:46.001026 27178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:37:46.005879 27178 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:37:46.006377 27178 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:37:46.013792 27178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e788b0943b3451e9381dc428ae94eed: Generated new TSK 0
I20260924 13:37:46.013974 27178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:37:46.040772 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38545
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:46.120643 27193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:46.120862 27193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:46.120901 27193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:46.122110 27193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:46.122169 27193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:46.122215 27193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:37:46.123695 27193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:46.123960 27193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:46.124193 27193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:46.129061 27199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:46.129061 27201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:46.129099 27198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:46.129318 27193 server_base.cc:1034] running on GCE node
I20260924 13:37:46.129539 27193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:46.129794 27193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:46.130941 27193 hybrid_clock.cc:648] HybridClock initialized: now 1790257066130929 us; error 28 us; skew 500 ppm
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:46.133759 27193 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:37:46.134258 27193 webserver.cc:460] Webserver started at http://127.25.240.65:39847/ using document root <none> and password file <none>
I20260924 13:37:46.134498 27193 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:46.134567 27193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:46.134719 27193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:46.135772 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "c74ec94ae74340fdbdcbc6003ad483ee"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "bc5caf477e43dbf781c06bddf2db1732"
server_key_iv: "723b4ead07a5329d80280449bb939ac2"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.136122 27193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "c74ec94ae74340fdbdcbc6003ad483ee"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "bc5caf477e43dbf781c06bddf2db1732"
server_key_iv: "723b4ead07a5329d80280449bb939ac2"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.137391 27193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:37:46.138177 27208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.138348 27193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:46.138443 27193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "c74ec94ae74340fdbdcbc6003ad483ee"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "bc5caf477e43dbf781c06bddf2db1732"
server_key_iv: "723b4ead07a5329d80280449bb939ac2"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.138528 27193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:46.149238 27193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:46.151304 27193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:46.151458 27193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:46.151681 27193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:37:46.151983 27193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:46.152038 27193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.152081 27193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:46.152119 27193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.158113 27193 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:37379
I20260924 13:37:46.158164 27321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:37379 every 8 connection(s)
I20260924 13:37:46.158469 27193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:46.165329 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 27193
I20260924 13:37:46.165452 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:37:46.165659 26561 external_mini_cluster.cc:1428] Setting key 9676856d5469f1ddabea41f7d8f13d18
I20260924 13:37:46.168136 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.159208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:38233 (local address 127.25.240.126:38545)
0924 13:37:46.159259 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.159275 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.159478 (+ 203us) server_negotiation.cc:407] Connection header received
0924 13:37:46.160442 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.160443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.160456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.160502 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:46.161142 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.161698 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.162722 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.162939 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.163043 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.165131 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:46.165133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:46.165133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:46.165140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:46.166748 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.167171 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.167172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.167172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.167234 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.167487 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.167488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.167488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.167603 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:46.167672 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.167950 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.168046 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:37:46.168375 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.159183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:38545 (local address 127.25.240.65:38233)
0924 13:37:46.159338 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:46.159381 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:46.160320 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:46.160569 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:46.160573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:46.160844 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:46.161032 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.161036 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.161792 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.161793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.162614 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.162616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.163050 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.163051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.163178 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.163643 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:46.163653 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:46.165028 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:46.166847 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:46.166854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:46.166856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:46.167084 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:46.167300 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:46.167301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:46.167301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:46.167413 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:46.167676 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:46.167683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:46.167819 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 13:37:46.167930 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 13:37:46.168083 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260924 13:37:46.168883 27322 heartbeater.cc:347] Connected to a master server at 127.25.240.126:38545
I20260924 13:37:46.168996 27322 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:46.169167 27322 heartbeater.cc:511] Master 127.25.240.126:38545 requested a full tablet report, sending...
I20260924 13:37:46.169767 27131 ts_manager.cc:194] Registered new tserver with Master: c74ec94ae74340fdbdcbc6003ad483ee (127.25.240.65:37379)
I20260924 13:37:46.170730 27131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:38233
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:37:46.190095 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38545
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:46.269760 27327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:46.269966 27327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:46.269990 27327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:46.271092 27327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:46.271131 27327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:46.271158 27327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:37:46.272598 27327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:46.272787 27327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:46.272956 27327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:37:46.277909 27327 server_base.cc:1034] running on GCE node
W20260924 13:37:46.278024 27332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:46.278024 27335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:46.278152 27333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:46.278479 27327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:46.278748 27327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:46.279925 27327 hybrid_clock.cc:648] HybridClock initialized: now 1790257066279922 us; error 37 us; skew 500 ppm
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:46.282450 27327 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:37:46.282920 27327 webserver.cc:460] Webserver started at http://127.25.240.66:43057/ using document root <none> and password file <none>
I20260924 13:37:46.283151 27327 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:46.283219 27327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:46.283368 27327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:46.284394 27327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "c094a6cc10ba40199926bce05828d38e"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "88020dcde46a282ddae469fe2f6d38d3"
server_key_iv: "d97918387496b05c42f6d3d264c0e2ac"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.284746 27327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "c094a6cc10ba40199926bce05828d38e"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "88020dcde46a282ddae469fe2f6d38d3"
server_key_iv: "d97918387496b05c42f6d3d264c0e2ac"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.286118 27327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:37:46.286813 27342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.286979 27327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.287081 27327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "c094a6cc10ba40199926bce05828d38e"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "88020dcde46a282ddae469fe2f6d38d3"
server_key_iv: "d97918387496b05c42f6d3d264c0e2ac"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.287150 27327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:46.302106 27327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:46.303941 27327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:46.304100 27327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:46.304317 27327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:37:46.304611 27327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:46.304668 27327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.304709 27327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:46.304736 27327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.310464 27327 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:42053
I20260924 13:37:46.310513 27455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:42053 every 8 connection(s)
I20260924 13:37:46.310799 27327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:37:46.314512 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 27327
I20260924 13:37:46.314585 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:37:46.314752 26561 external_mini_cluster.cc:1428] Setting key a22827e7ce400207f0ce43d4054712f9
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:46.320549 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.311510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:36047 (local address 127.25.240.126:38545)
0924 13:37:46.311583 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.311585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.311739 (+ 154us) server_negotiation.cc:407] Connection header received
0924 13:37:46.312645 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.312646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.312655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.312712 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:46.313291 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.313891 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.315020 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.315282 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.315355 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.317611 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:46.317612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:46.317613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:46.317619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:46.319130 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.319601 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.319602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.319603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.319653 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.319934 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.319935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.319935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.320030 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:46.320078 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.320424 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.320488 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 13:37:46.320847 27458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.311456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:38545 (local address 127.25.240.66:36047)
0924 13:37:46.311611 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:46.311651 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:46.312552 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:46.312790 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:46.312793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:46.313036 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:46.313218 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.313221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.313967 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.313968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.314898 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.314900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.315373 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.315373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.315512 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.316023 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:46.316031 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:46.317486 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:46.319214 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:46.319222 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:46.319225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:46.319526 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:46.319719 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:46.319720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:46.319720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:46.319865 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:46.320095 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:46.320101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:46.320270 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 13:37:46.320402 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 13:37:46.320576 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260924 13:37:46.321403 27456 heartbeater.cc:347] Connected to a master server at 127.25.240.126:38545
I20260924 13:37:46.321501 27456 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:46.321723 27456 heartbeater.cc:511] Master 127.25.240.126:38545 requested a full tablet report, sending...
I20260924 13:37:46.322127 27131 ts_manager.cc:194] Registered new tserver with Master: c094a6cc10ba40199926bce05828d38e (127.25.240.66:42053)
I20260924 13:37:46.322492 27131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:36047
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:37:46.341400 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38545
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:46.435235 27461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:46.435534 27461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:46.435577 27461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:46.437770 27461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:46.437832 27461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:46.437882 27461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:37:46.440567 27461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38545
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:46.440872 27461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:46.441138 27461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:46.446676 27467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:46.446684 27466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:46.446739 27461 server_base.cc:1034] running on GCE node
W20260924 13:37:46.446740 27469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:46.447085 27461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:46.447337 27461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:46.448529 27461 hybrid_clock.cc:648] HybridClock initialized: now 1790257066448519 us; error 42 us; skew 500 ppm
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:46.451160 27461 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:37:46.451576 27461 webserver.cc:460] Webserver started at http://127.25.240.67:36395/ using document root <none> and password file <none>
I20260924 13:37:46.451840 27461 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:46.451905 27461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:46.452044 27461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:46.453085 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "2c183de6af374e6c822f3f51fe924424"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "60c5b32d83d0d8dd97dff52e5a46ccd7"
server_key_iv: "1b1b8106cbee110483c8257d76565d92"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.453444 27461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "2c183de6af374e6c822f3f51fe924424"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "60c5b32d83d0d8dd97dff52e5a46ccd7"
server_key_iv: "1b1b8106cbee110483c8257d76565d92"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.454785 27461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:37:46.455549 27476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.455730 27461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:37:46.455818 27461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "2c183de6af374e6c822f3f51fe924424"
format_stamp: "Formatted at 2026-09-24 13:37:46 on dist-test-slave-m5z1"
server_key: "60c5b32d83d0d8dd97dff52e5a46ccd7"
server_key_iv: "1b1b8106cbee110483c8257d76565d92"
server_key_version: "encryptionkey@0"
I20260924 13:37:46.455870 27461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:46.477734 27461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:46.480078 27461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:46.480229 27461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:46.480480 27461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:37:46.480801 27461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:46.480855 27461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.480898 27461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:46.480926 27461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:46.486672 27461 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:39701
I20260924 13:37:46.486724 27589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:39701 every 8 connection(s)
I20260924 13:37:46.487071 27461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:46.495918 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.487823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:36415 (local address 127.25.240.126:38545)
0924 13:37:46.487891 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.487893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.488102 (+ 209us) server_negotiation.cc:407] Connection header received
0924 13:37:46.489002 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.489003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.489013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.489094 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:46.489664 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.490275 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.491366 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.491557 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.491610 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.493653 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:46.493655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:46.493655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:46.493661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:46.494830 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.495174 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.495175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.495175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.495224 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.495444 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.495445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.495445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.495541 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:46.495591 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.495800 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.495856 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:37:46.496062 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.487805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:38545 (local address 127.25.240.67:36415)
0924 13:37:46.487984 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:46.488019 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:46.488914 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:46.489145 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:46.489148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:46.489395 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:46.489603 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.489606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.490354 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.490355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.491290 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.491292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.491623 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.491624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.491715 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.492132 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:46.492139 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:46.493487 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:46.494899 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:46.494904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:46.494905 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:46.495112 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:46.495279 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:46.495279 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:46.495279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:46.495391 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:46.495593 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:46.495597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:46.495697 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 13:37:46.495773 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:37:46.495879 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260924 13:37:46.496353 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 27461
I20260924 13:37:46.496454 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:37:46.496644 26561 external_mini_cluster.cc:1428] Setting key 4aef9907a9faf2f7bdf5df04706ce6fd
I20260924 13:37:46.496650 27590 heartbeater.cc:347] Connected to a master server at 127.25.240.126:38545
I20260924 13:37:46.496732 27590 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:46.496923 27590 heartbeater.cc:511] Master 127.25.240.126:38545 requested a full tablet report, sending...
I20260924 13:37:46.497336 27131 ts_manager.cc:194] Registered new tserver with Master: 2c183de6af374e6c822f3f51fe924424 (127.25.240.67:39701)
I20260924 13:37:46.497740 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:37:46.502070 27131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:36415
I20260924 13:37:46.507689 27170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.501861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.25.240.126:38545)
0924 13:37:46.501895 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.501897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.501905 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:37:46.501931 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.501932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.501941 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.501999 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:46.502697 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.503154 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.503826 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.504015 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.504072 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.505331 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:46.505332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:46.505332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:46.505337 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:46.506645 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.506994 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.506995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.506996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.507044 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.507283 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.507283 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.507283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.507405 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:46.507477 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.507573 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.507630 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:37:46.509780 27131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:37:46.511356 27131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:46.517823 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.513638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37688 (local address 127.25.240.67:39701)
0924 13:37:46.513840 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.513845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.513886 (+ 41us) server_negotiation.cc:407] Connection header received
0924 13:37:46.513928 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.513930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.513947 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.514028 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:46.514252 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.515059 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.516563 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.517290 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.517374 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.517435 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.517685 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.517730 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260924 13:37:46.517822 27598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.513528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:39701 (local address 127.0.0.1:37688)
0924 13:37:46.513712 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:46.513751 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:46.513813 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:46.514088 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:46.514091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:46.514093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:46.514161 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.514163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.515200 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.515201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.516489 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.516491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.517473 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.517474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.517580 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.517593 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:37:46.517658 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 13:37:46.517730 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":79,"threads_started":1}
I20260924 13:37:46.517827 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.513742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:42053 (local address 127.0.0.1:60566)
0924 13:37:46.513963 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:46.513973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:46.514012 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:46.514483 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:46.514484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:46.514486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:46.514526 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.514528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.515733 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.515735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.516623 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.516625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.517469 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.517470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.517546 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.517592 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:37:46.517666 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 13:37:46.517737 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":24,"thread_start_us":65,"threads_started":1}
I20260924 13:37:46.518262 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.513958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.25.240.66:42053)
0924 13:37:46.514127 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.514131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.514139 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:37:46.514298 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.514301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.514327 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.514403 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:46.514600 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.515616 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.516710 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.517394 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.517458 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.517506 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.517955 (+ 449us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.518192 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":70,"threads_started":1}
I20260924 13:37:46.518590 27524 tablet_service.cc:1467] Processing CreateTablet for tablet 4d30565904a64f04877f8bea31495bf0 (DEFAULT_TABLE table=test-table [id=ab41fe3a2c2544a9bac93ad59b027f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:46.518821 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.513528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:37379 (local address 127.0.0.1:60954)
0924 13:37:46.513909 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:46.513918 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:46.513963 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:46.514194 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:46.514195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:46.514198 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:46.514255 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.514258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.516450 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.516451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.517184 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.517192 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.518574 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.518575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.518641 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.518645 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:46.518706 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:37:46.518745 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260924 13:37:46.518847 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.513681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.25.240.65:37379)
0924 13:37:46.513842 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.513846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.514031 (+ 185us) server_negotiation.cc:407] Connection header received
0924 13:37:46.514079 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.514082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.514096 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.514175 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:46.514351 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.515471 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.517299 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.518500 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.518574 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.518630 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.518730 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.518770 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":65,"threads_started":1}
I20260924 13:37:46.518960 27524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d30565904a64f04877f8bea31495bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:46.518905 27390 tablet_service.cc:1467] Processing CreateTablet for tablet 4d30565904a64f04877f8bea31495bf0 (DEFAULT_TABLE table=test-table [id=ab41fe3a2c2544a9bac93ad59b027f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:46.519246 27390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d30565904a64f04877f8bea31495bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:46.519472 27256 tablet_service.cc:1467] Processing CreateTablet for tablet 4d30565904a64f04877f8bea31495bf0 (DEFAULT_TABLE table=test-table [id=ab41fe3a2c2544a9bac93ad59b027f63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:46.519749 27256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d30565904a64f04877f8bea31495bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:46.521667 27604 tablet_bootstrap.cc:492] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e: Bootstrap starting.
I20260924 13:37:46.522257 27605 tablet_bootstrap.cc:492] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424: Bootstrap starting.
I20260924 13:37:46.522323 27604 tablet_bootstrap.cc:654] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:46.522956 27606 tablet_bootstrap.cc:492] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee: Bootstrap starting.
I20260924 13:37:46.523134 27605 tablet_bootstrap.cc:654] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:46.522941 27604 log.cc:824] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:46.523593 27606 tablet_bootstrap.cc:654] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:46.523710 27605 log.cc:824] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:46.523948 27604 tablet_bootstrap.cc:492] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e: No bootstrap required, opened a new log
I20260924 13:37:46.524014 27604 ts_tablet_manager.cc:1397] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:37:46.524174 27606 log.cc:824] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:46.524477 27605 tablet_bootstrap.cc:492] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424: No bootstrap required, opened a new log
I20260924 13:37:46.524545 27605 ts_tablet_manager.cc:1397] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:37:46.524861 27606 tablet_bootstrap.cc:492] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee: No bootstrap required, opened a new log
I20260924 13:37:46.524945 27606 ts_tablet_manager.cc:1397] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:37:46.526034 27604 raft_consensus.cc:348] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.526227 27604 raft_consensus.cc:374] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:46.526263 27604 raft_consensus.cc:729] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c094a6cc10ba40199926bce05828d38e, State: Initialized, Role: FOLLOWER
I20260924 13:37:46.526401 27604 consensus_queue.cc:260] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.526648 27604 ts_tablet_manager.cc:1428] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 13:37:46.526749 27456 heartbeater.cc:503] Master 127.25.240.126:38545 was elected leader, sending a full tablet report...
I20260924 13:37:46.526989 27606 raft_consensus.cc:348] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.527158 27606 raft_consensus.cc:374] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:46.527194 27606 raft_consensus.cc:729] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74ec94ae74340fdbdcbc6003ad483ee, State: Initialized, Role: FOLLOWER
I20260924 13:37:46.527294 27606 consensus_queue.cc:260] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.527339 27605 raft_consensus.cc:348] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.527488 27605 raft_consensus.cc:374] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:46.527575 27606 ts_tablet_manager.cc:1428] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:37:46.527700 27322 heartbeater.cc:503] Master 127.25.240.126:38545 was elected leader, sending a full tablet report...
I20260924 13:37:46.527524 27605 raft_consensus.cc:729] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c183de6af374e6c822f3f51fe924424, State: Initialized, Role: FOLLOWER
I20260924 13:37:46.528107 27605 consensus_queue.cc:260] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.528409 27590 heartbeater.cc:503] Master 127.25.240.126:38545 was elected leader, sending a full tablet report...
I20260924 13:37:46.528389 27605 ts_tablet_manager.cc:1428] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 13:37:46.554834 27611 raft_consensus.cc:482] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:46.554941 27611 raft_consensus.cc:504] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.555382 27611 leader_election.cc:288] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c183de6af374e6c822f3f51fe924424 (127.25.240.67:39701), c094a6cc10ba40199926bce05828d38e (127.25.240.66:42053)
I20260924 13:37:46.559345 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.555596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:50137 (local address 127.25.240.67:39701)
0924 13:37:46.555653 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.555655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.555844 (+ 189us) server_negotiation.cc:407] Connection header received
0924 13:37:46.555888 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.555889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.555896 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.555935 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:46.556116 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.556789 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.558387 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.558977 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.559054 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.559062 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.559251 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.559295 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:37:46.559407 27324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.555538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:42053 (local address 127.25.240.65:40737)
0924 13:37:46.555600 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:46.555612 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:46.555652 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:46.555817 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:46.555819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:46.555820 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:46.555896 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.555898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.556909 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.556910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.558288 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.558291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.559062 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.559063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.559152 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.559155 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:37:46.559209 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 13:37:46.559256 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":91}
I20260924 13:37:46.559499 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.555548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:39701 (local address 127.25.240.65:50137)
0924 13:37:46.555722 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:46.555730 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:46.555827 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:46.555960 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:46.555961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:46.555963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:46.556046 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.556048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.556908 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.556910 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.558288 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:46.558291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.559062 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.559063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:46.559142 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.559147 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:37:46.559209 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 13:37:46.559256 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":46,"thread_start_us":99,"threads_started":1}
I20260924 13:37:46.559602 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.555632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:40737 (local address 127.25.240.66:42053)
0924 13:37:46.555676 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.555678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.555703 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:37:46.555735 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.555736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.555743 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.555783 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:46.555996 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.556752 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.558400 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.558985 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.559054 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.559062 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.559465 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.559517 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:37:46.559926 27410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d30565904a64f04877f8bea31495bf0" candidate_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c094a6cc10ba40199926bce05828d38e" is_pre_election: true
I20260924 13:37:46.560091 27410 raft_consensus.cc:2454] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74ec94ae74340fdbdcbc6003ad483ee in term 0.
I20260924 13:37:46.560092 27544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d30565904a64f04877f8bea31495bf0" candidate_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c183de6af374e6c822f3f51fe924424" is_pre_election: true
I20260924 13:37:46.560243 27544 raft_consensus.cc:2454] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74ec94ae74340fdbdcbc6003ad483ee in term 0.
I20260924 13:37:46.560323 27211 leader_election.cc:302] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c094a6cc10ba40199926bce05828d38e, c74ec94ae74340fdbdcbc6003ad483ee; no voters:
I20260924 13:37:46.560454 27611 raft_consensus.cc:2781] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:46.560514 27611 raft_consensus.cc:482] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:46.560562 27611 raft_consensus.cc:3037] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 0 FOLLOWER]: Advancing to term 1
W20260924 13:37:46.561311 27457 tablet.cc:2369] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:37:46.619699 27611 raft_consensus.cc:504] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.619943 27611 leader_election.cc:288] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [CANDIDATE]: Term 1 election: Requested vote from peers 2c183de6af374e6c822f3f51fe924424 (127.25.240.67:39701), c094a6cc10ba40199926bce05828d38e (127.25.240.66:42053)
I20260924 13:37:46.620146 27544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d30565904a64f04877f8bea31495bf0" candidate_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c183de6af374e6c822f3f51fe924424"
I20260924 13:37:46.620241 27410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d30565904a64f04877f8bea31495bf0" candidate_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c094a6cc10ba40199926bce05828d38e"
I20260924 13:37:46.620267 27544 raft_consensus.cc:3037] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:46.620332 27410 raft_consensus.cc:3037] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:46.621260 27410 raft_consensus.cc:2454] T 4d30565904a64f04877f8bea31495bf0 P c094a6cc10ba40199926bce05828d38e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74ec94ae74340fdbdcbc6003ad483ee in term 1.
I20260924 13:37:46.621260 27544 raft_consensus.cc:2454] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74ec94ae74340fdbdcbc6003ad483ee in term 1.
I20260924 13:37:46.621475 27210 leader_election.cc:302] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c183de6af374e6c822f3f51fe924424, c74ec94ae74340fdbdcbc6003ad483ee; no voters:
I20260924 13:37:46.621593 27611 raft_consensus.cc:2781] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:46.621775 27611 raft_consensus.cc:686] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [term 1 LEADER]: Becoming Leader. State: Replica: c74ec94ae74340fdbdcbc6003ad483ee, State: Running, Role: LEADER
I20260924 13:37:46.621906 27611 consensus_queue.cc:237] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } }
I20260924 13:37:46.622815 27130 catalog_manager.cc:5495] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee reported cstate change: term changed from 0 to 1, leader changed from <none> to c74ec94ae74340fdbdcbc6003ad483ee (127.25.240.65). New cstate: current_term: 1 leader_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74ec94ae74340fdbdcbc6003ad483ee" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 37379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c183de6af374e6c822f3f51fe924424" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c094a6cc10ba40199926bce05828d38e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42053 } health_report { overall_health: UNKNOWN } } }
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:37:46.639863 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.632553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.25.240.65:37379)
0924 13:37:46.632633 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.632635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.632747 (+ 112us) server_negotiation.cc:407] Connection header received
0924 13:37:46.632861 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.632862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.632873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.632955 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:46.633523 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.634039 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.634820 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.635018 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.635091 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.637339 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:46.637350 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:46.637350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:46.637363 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:46.638812 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.639142 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.639145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.639145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.639225 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.639407 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.639408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.639408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.639548 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:46.639664 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.639712 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.639771 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 13:37:46.659205 27323 tablet.cc:2369] T 4d30565904a64f04877f8bea31495bf0 P c74ec94ae74340fdbdcbc6003ad483ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:37:46.733117 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.725028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.25.240.65:37379)
0924 13:37:46.725083 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.725085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.725168 (+ 83us) server_negotiation.cc:407] Connection header received
0924 13:37:46.725286 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.725287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.725299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.725360 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:46.726100 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.726953 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.727801 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.728183 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.728257 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.730697 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:46.730698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:46.730698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:46.730705 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:46.731983 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.732308 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.732309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.732309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.732361 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.732553 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.732554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.732554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.732658 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:46.732751 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.732889 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.732970 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 13:37:46.737654 27591 tablet.cc:2369] T 4d30565904a64f04877f8bea31495bf0 P 2c183de6af374e6c822f3f51fe924424: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:37:46 dist-test-slave-m5z1 krb5kdc[27086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:37:46.833061 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:46.825232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.25.240.65:37379)
0924 13:37:46.825309 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:46.825311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:46.825432 (+ 121us) server_negotiation.cc:407] Connection header received
0924 13:37:46.825511 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:46.825512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:46.825522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:46.825642 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:46.826406 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.826881 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.828055 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:46.828239 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:46.828309 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:46.830840 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:46.830841 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:46.830842 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:46.830848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:46.832007 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.832375 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.832376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.832377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.832449 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:46.832609 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:46.832610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:46.832610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:46.832698 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:46.832772 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:46.832937 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 13:37:46.832997 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 13:37:46.833608 27236 tablet_service.cc:2908] Rejecting scan request for tablet 4d30565904a64f04877f8bea31495bf0: Uninitialized: clean time has not yet been initialized
I20260924 13:37:46.834458 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 27193
I20260924 13:37:46.839571 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 27327
I20260924 13:37:46.843991 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 27461
I20260924 13:37:46.848147 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 27099
2026-09-24T13:37:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2141 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:37:48 dist-test-slave-m5z1 krb5kdc[27628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:37:48 dist-test-slave-m5z1 krb5kdc[27628](info): set up 2 sockets
Sep 24 13:37:48 dist-test-slave-m5z1 krb5kdc[27628](info): commencing operation
krb5kdc: starting...
W20260924 13:37:49.583447 26561 mini_kdc.cc:121] Time spent starting KDC: real 2.714s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 13:37:49.800314 26561 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.124s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 13:37:49.939826 26561 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.139s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/test-user.keytab.
W20260924 13:37:50.545370 26561 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.605s user 0.000s sys 0.002s
Sep 24 13:37:50 dist-test-slave-m5z1 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:37:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:37:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
W20260924 13:37:51.532531 26561 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.25.240.126: real 0.967s user 0.000s sys 0.003s
I20260924 13:37:51.532883 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:33565
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:33565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:51.616264 27641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:51.616468 27641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:51.616490 27641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:51.617719 27641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:37:51.617759 27641 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 13:37:51.617772 27641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:51.617786 27641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:51.617798 27641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:37:51.617810 27641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:37:51.619200 27641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:33565
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:33565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:51.619383 27641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:51.619542 27641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:51.624261 27649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:51.624302 27646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:51.624264 27647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:51.624509 27641 server_base.cc:1034] running on GCE node
I20260924 13:37:51.624735 27641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:51.625022 27641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:51.626191 27641 hybrid_clock.cc:648] HybridClock initialized: now 1790257071626183 us; error 40 us; skew 500 ppm
Sep 24 13:37:51 dist-test-slave-m5z1 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257071, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:51.635727 27641 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:37:51.636108 27641 webserver.cc:460] Webserver started at http://127.25.240.126:44035/ using document root <none> and password file <none>
I20260924 13:37:51.636286 27641 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:51.636323 27641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:51.636418 27641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:51.637534 27641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "5c5cbb650ad54659bb67d68e1dfc94fa"
format_stamp: "Formatted at 2026-09-24 13:37:51 on dist-test-slave-m5z1"
server_key: "c2161043a248c9fef0243d2cedc2fab6"
server_key_iv: "8ee11da72b1dcbdeb47184c019b3a868"
server_key_version: "encryptionkey@0"
I20260924 13:37:51.637953 27641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "5c5cbb650ad54659bb67d68e1dfc94fa"
format_stamp: "Formatted at 2026-09-24 13:37:51 on dist-test-slave-m5z1"
server_key: "c2161043a248c9fef0243d2cedc2fab6"
server_key_iv: "8ee11da72b1dcbdeb47184c019b3a868"
server_key_version: "encryptionkey@0"
I20260924 13:37:51.639271 27641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:51.640017 27656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:51.640208 27641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:51.640293 27641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "5c5cbb650ad54659bb67d68e1dfc94fa"
format_stamp: "Formatted at 2026-09-24 13:37:51 on dist-test-slave-m5z1"
server_key: "c2161043a248c9fef0243d2cedc2fab6"
server_key_iv: "8ee11da72b1dcbdeb47184c019b3a868"
server_key_version: "encryptionkey@0"
I20260924 13:37:51.640347 27641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:51.655488 27641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:51.657769 27641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:51.657881 27641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:51.662281 27641 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:33565
I20260924 13:37:51.662374 27718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:33565 every 8 connection(s)
I20260924 13:37:51.662683 27641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:37:51.663364 27719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:51.665714 27719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa: Bootstrap starting.
I20260924 13:37:51.666303 27719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:51.666594 27719 log.cc:824] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:51.667354 27719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa: No bootstrap required, opened a new log
I20260924 13:37:51.667770 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 27641
I20260924 13:37:51.667860 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:37:51.667979 26561 external_mini_cluster.cc:1428] Setting key e83c3a698862e3d4da0e1706c7e8d09c
I20260924 13:37:51.670207 27719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cbb650ad54659bb67d68e1dfc94fa" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33565 } }
I20260924 13:37:51.670364 27719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:51.670428 27719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c5cbb650ad54659bb67d68e1dfc94fa, State: Initialized, Role: FOLLOWER
I20260924 13:37:51.670590 27719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cbb650ad54659bb67d68e1dfc94fa" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33565 } }
I20260924 13:37:51.670681 27719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:37:51.670727 27719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:37:51.670778 27719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 0 FOLLOWER]: Advancing to term 1
Sep 24 13:37:51 dist-test-slave-m5z1 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:51.678697 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:51.668505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.25.240.126:33565)
0924 13:37:51.668690 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:51.668700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:51.668737 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:37:51.670198 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:51.670205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:51.670259 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:51.670412 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:51.671171 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:51.672050 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:51.672813 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:51.673217 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:51.673341 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:51.675571 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:51.675575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:51.675576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:51.675595 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:51.677192 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:51.677593 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:51.677596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:51.677597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:51.677686 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:51.677898 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:51.677898 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:51.677899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:51.678131 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:51.678281 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:51.678400 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 13:37:51.678542 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":76,"threads_started":1}
I20260924 13:37:51.729456 27719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cbb650ad54659bb67d68e1dfc94fa" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33565 } }
I20260924 13:37:51.729677 27719 leader_election.cc:302] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c5cbb650ad54659bb67d68e1dfc94fa; no voters:
I20260924 13:37:51.729904 27719 leader_election.cc:288] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:37:51.729996 27724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:51.730196 27719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:37:51.730204 27724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [term 1 LEADER]: Becoming Leader. State: Replica: 5c5cbb650ad54659bb67d68e1dfc94fa, State: Running, Role: LEADER
I20260924 13:37:51.730494 27724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cbb650ad54659bb67d68e1dfc94fa" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33565 } }
I20260924 13:37:51.730909 27725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c5cbb650ad54659bb67d68e1dfc94fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cbb650ad54659bb67d68e1dfc94fa" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33565 } } }
I20260924 13:37:51.730997 27725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [sys.catalog]: This master's current role is: LEADER
I20260924 13:37:51.731419 27726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c5cbb650ad54659bb67d68e1dfc94fa. Latest consensus state: current_term: 1 leader_uuid: "5c5cbb650ad54659bb67d68e1dfc94fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c5cbb650ad54659bb67d68e1dfc94fa" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33565 } } }
I20260924 13:37:51.731501 27726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa [sys.catalog]: This master's current role is: LEADER
W20260924 13:37:51.732398 27740 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:37:51.732456 27740 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:37:51.732515 27733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:37:51.732697 27733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:37:51.734326 27733 catalog_manager.cc:1347] Generated new cluster ID: f3982e9536f44d58bc5b2d59d165f589
I20260924 13:37:51.734385 27733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:37:51.743188 27733 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:37:51.743762 27733 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:37:51.749949 27733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa: Generated new TSK 0
I20260924 13:37:51.750156 27733 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
W20260924 13:37:51.988134 26561 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.25.240.65: real 0.228s user 0.000s sys 0.003s
I20260924 13:37:51.988458 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33565
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:52.067416 27745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:52.067677 27745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:52.067714 27745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:52.068902 27745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:52.068970 27745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:52.069002 27745 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:37:52.069028 27745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:37:52.070480 27745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:52.070750 27745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:52.070976 27745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:52.075850 27751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:52.075881 27750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:52.075850 27753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:52.076205 27745 server_base.cc:1034] running on GCE node
I20260924 13:37:52.076401 27745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:52.076669 27745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:52.077824 27745 hybrid_clock.cc:648] HybridClock initialized: now 1790257072077808 us; error 26 us; skew 500 ppm
Sep 24 13:37:52 dist-test-slave-m5z1 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:52.080196 27745 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:37:52.080621 27745 webserver.cc:460] Webserver started at http://127.25.240.65:36993/ using document root <none> and password file <none>
I20260924 13:37:52.080856 27745 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:52.080933 27745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:52.081084 27745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:52.082185 27745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "d03b768e9b0e40d3a046a08745e4a22d"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "6a0be5eb643db10e1b7cca318f421c5f"
server_key_iv: "b87a587217b152f838462bf926e1f105"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.082540 27745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "d03b768e9b0e40d3a046a08745e4a22d"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "6a0be5eb643db10e1b7cca318f421c5f"
server_key_iv: "b87a587217b152f838462bf926e1f105"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.083832 27745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:37:52.084533 27760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:52.084697 27745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:52.084800 27745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "d03b768e9b0e40d3a046a08745e4a22d"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "6a0be5eb643db10e1b7cca318f421c5f"
server_key_iv: "b87a587217b152f838462bf926e1f105"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.084872 27745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:52.093360 27745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:52.095134 27745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:52.095287 27745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:52.095763 27745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:52.095826 27745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:52.095870 27745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:52.095894 27745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:37:52 dist-test-slave-m5z1 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:52.103371 27745 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:41237
I20260924 13:37:52.103451 27875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:41237 every 8 connection(s)
I20260924 13:37:52.103767 27745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:37:52.105461 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.096253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:33227 (local address 127.25.240.126:33565)
0924 13:37:52.096326 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.096328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.096480 (+ 152us) server_negotiation.cc:407] Connection header received
0924 13:37:52.096944 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.096946 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.096959 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.097039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:52.097872 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.098722 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.099822 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.100170 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.100240 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.102593 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:52.102595 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:52.102595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:52.102605 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:52.104372 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.104727 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.104728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.104728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.104777 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.104946 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.104947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.104947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.105055 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:52.105105 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.105332 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.105399 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:37:52.105619 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.096208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33565 (local address 127.25.240.65:33227)
0924 13:37:52.096356 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.096394 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.096849 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.097102 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.097106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.097451 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:52.097759 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.097765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.098823 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.098824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.099727 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.099729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.100259 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.100260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.100384 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.100875 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:52.100887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:52.102490 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:52.104442 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.104446 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.104448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.104660 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.104817 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.104817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.104818 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.104897 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.105114 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:52.105119 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:52.105223 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.105304 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.105414 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260924 13:37:52.106132 27876 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33565
I20260924 13:37:52.106230 27876 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:52.106397 27876 heartbeater.cc:511] Master 127.25.240.126:33565 requested a full tablet report, sending...
I20260924 13:37:52.106902 27672 ts_manager.cc:194] Registered new tserver with Master: d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237)
I20260924 13:37:52.107816 27672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:33227
I20260924 13:37:52.111987 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.107202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:55873 (local address 127.25.240.126:33565)
0924 13:37:52.107245 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.107246 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.107319 (+ 73us) server_negotiation.cc:407] Connection header received
0924 13:37:52.107383 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.107384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.107394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.107433 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:52.107774 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.108371 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.109110 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.109279 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.109331 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.110143 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:52.110143 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:52.110144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:52.110148 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:52.111231 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.111518 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.111519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.111519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.111565 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.111678 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.111678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.111678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.111757 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:52.111798 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.111879 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.111935 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:37:52.111994 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.107124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33565 (local address 127.25.240.65:55873)
0924 13:37:52.107260 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.107273 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.107331 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.107445 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.107446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.107645 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:52.107708 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.107710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.108454 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.108455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.109039 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.109041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.109331 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.109332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.109380 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.109805 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:52.109810 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:52.110074 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:52.111299 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.111300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.111300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.111452 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.111605 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.111605 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.111605 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.111645 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.111790 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:52.111790 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:52.111821 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.111865 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.111930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260924 13:37:52.113127 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 27745
I20260924 13:37:52.113233 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:37:52.113435 26561 external_mini_cluster.cc:1428] Setting key 4021cfc14e179b243156e01ba5683675
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:37:52.136195 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33565
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:52.215282 27885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:52.215502 27885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:52.215538 27885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:52.216665 27885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:52.216727 27885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:52.216763 27885 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:37:52.216820 27885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:37:52.218279 27885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:52.218551 27885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:52.218768 27885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:52.223501 27891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:52.223541 27893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:52.223526 27890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:52.223824 27885 server_base.cc:1034] running on GCE node
I20260924 13:37:52.224052 27885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:52.224278 27885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:52.225422 27885 hybrid_clock.cc:648] HybridClock initialized: now 1790257072225401 us; error 53 us; skew 500 ppm
Sep 24 13:37:52 dist-test-slave-m5z1 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:52.227766 27885 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:37:52.228214 27885 webserver.cc:460] Webserver started at http://127.25.240.66:43417/ using document root <none> and password file <none>
I20260924 13:37:52.228386 27885 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:52.228425 27885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:52.228507 27885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:52.229435 27885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "96cc2d35e9004341b0d7f3079333ce1b"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "2a58f97d1974ff1b3b073b442e213b95"
server_key_iv: "cf94eb8c2b5c23b0b4d82f3b71c32db4"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.229826 27885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "96cc2d35e9004341b0d7f3079333ce1b"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "2a58f97d1974ff1b3b073b442e213b95"
server_key_iv: "cf94eb8c2b5c23b0b4d82f3b71c32db4"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.231248 27885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:52.231963 27900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:52.232164 27885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:52.232259 27885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "96cc2d35e9004341b0d7f3079333ce1b"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "2a58f97d1974ff1b3b073b442e213b95"
server_key_iv: "cf94eb8c2b5c23b0b4d82f3b71c32db4"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.232347 27885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:52.251794 27885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:52.253854 27885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:52.254004 27885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:52.254488 27885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:52.254571 27885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:52.254622 27885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:52.254652 27885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:52.261240 27885 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:38673
I20260924 13:37:52.261308 28015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:38673 every 8 connection(s)
Sep 24 13:37:52 dist-test-slave-m5z1 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:52.261651 27885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:37:52.264993 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.255043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:36759 (local address 127.25.240.126:33565)
0924 13:37:52.255112 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.255114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.255309 (+ 195us) server_negotiation.cc:407] Connection header received
0924 13:37:52.255767 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.255768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.255780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.255860 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:52.256720 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.257581 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.259107 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.259425 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.259519 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.262151 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:52.262153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:52.262153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:52.262161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:52.263618 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.264034 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.264035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.264035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.264098 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.264358 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.264359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.264359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.264477 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:52.264548 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.264819 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.264901 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:37:52.265192 27909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.255014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33565 (local address 127.25.240.66:36759)
0924 13:37:52.255182 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.255222 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.255671 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.255926 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.255930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.256324 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:52.256609 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.256616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.257676 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.257677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.259026 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.259029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.259528 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.259529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.259658 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.260174 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:52.260186 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:52.262061 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:52.263731 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.263737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.263739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.263956 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.264175 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.264176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.264176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.264283 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.264550 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:52.264556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:52.264690 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.264797 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.264949 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260924 13:37:52.265695 28016 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33565
I20260924 13:37:52.265779 28016 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:52.265964 28016 heartbeater.cc:511] Master 127.25.240.126:33565 requested a full tablet report, sending...
I20260924 13:37:52.266337 27672 ts_manager.cc:194] Registered new tserver with Master: 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673)
I20260924 13:37:52.266827 27672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:36759
I20260924 13:37:52.270919 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 27885
I20260924 13:37:52.271016 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:37:52.271180 26561 external_mini_cluster.cc:1428] Setting key 0072d357335ed531112d116e040b11bf
I20260924 13:37:52.272122 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.266678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:41605 (local address 127.25.240.126:33565)
0924 13:37:52.266715 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.266717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.266838 (+ 121us) server_negotiation.cc:407] Connection header received
0924 13:37:52.266917 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.266918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.266929 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.266982 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:52.267303 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.267753 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.268421 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.268599 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.268663 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.269454 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:52.269455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:52.269455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:52.269460 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:52.270907 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.271314 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.271315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.271316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.271379 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.271582 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.271582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.271583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.271728 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:52.271806 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.271961 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.272034 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:37:52.272137 28022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.266607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33565 (local address 127.25.240.66:41605)
0924 13:37:52.266772 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.266782 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.266844 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.266987 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.266988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.267151 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:52.267226 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.267228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.267842 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.267842 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.268314 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.268315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.268673 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.268673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.268725 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.269086 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:52.269090 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:52.269339 (+ 249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:52.271011 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.271012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.271012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.271222 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.271446 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.271447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.271447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.271504 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.271797 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:52.271798 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:52.271844 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.271937 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.272035 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:37:52.296676 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33565
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:52.376205 28025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:52.376405 28025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:52.376428 28025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:52.377611 28025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:52.377650 28025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:52.377671 28025 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:37:52.377728 28025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:37:52.379128 28025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:33565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:52.379369 28025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:52.379606 28025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:52.384727 28033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:52.384727 28030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:52.384876 28025 server_base.cc:1034] running on GCE node
W20260924 13:37:52.384756 28031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:52.385200 28025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:52.385401 28025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:52.386540 28025 hybrid_clock.cc:648] HybridClock initialized: now 1790257072386532 us; error 32 us; skew 500 ppm
Sep 24 13:37:52 dist-test-slave-m5z1 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:52.389039 28025 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:37:52.389421 28025 webserver.cc:460] Webserver started at http://127.25.240.67:42459/ using document root <none> and password file <none>
I20260924 13:37:52.389688 28025 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:52.389737 28025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:52.389832 28025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:52.390825 28025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "dfb3688b8e5d4660b712c0b263973f58"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "219c40cfd3a79dec6153a18e5b2f605c"
server_key_iv: "5e3114968b7a2b6bff0ec98987521770"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.391196 28025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "dfb3688b8e5d4660b712c0b263973f58"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "219c40cfd3a79dec6153a18e5b2f605c"
server_key_iv: "5e3114968b7a2b6bff0ec98987521770"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.392452 28025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:37:52.393119 28040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:52.393298 28025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:37:52.393362 28025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "dfb3688b8e5d4660b712c0b263973f58"
format_stamp: "Formatted at 2026-09-24 13:37:52 on dist-test-slave-m5z1"
server_key: "219c40cfd3a79dec6153a18e5b2f605c"
server_key_iv: "5e3114968b7a2b6bff0ec98987521770"
server_key_version: "encryptionkey@0"
I20260924 13:37:52.393455 28025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:52.413753 28025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:52.415599 28025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:52.415745 28025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:52.416286 28025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:52.416360 28025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:52.416397 28025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:52.416428 28025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:37:52 dist-test-slave-m5z1 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:52.423635 28025 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:39935
I20260924 13:37:52.423717 28155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:39935 every 8 connection(s)
I20260924 13:37:52.424012 28025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:37:52.426476 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.416803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:40175 (local address 127.25.240.126:33565)
0924 13:37:52.416860 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.416862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.417090 (+ 228us) server_negotiation.cc:407] Connection header received
0924 13:37:52.417679 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.417681 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.417697 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.417762 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:52.418590 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.419423 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.420800 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.421164 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.421275 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.423714 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:52.423715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:52.423715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:52.423723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:52.425244 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.425674 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.425675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.425675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.425739 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.425955 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.425955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.425955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.426066 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:52.426114 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.426350 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.426413 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 13:37:52.426654 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.416789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33565 (local address 127.25.240.67:40175)
0924 13:37:52.416963 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.416999 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.417515 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.417838 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.417842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.418200 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:52.418489 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.418497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.419506 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.419507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.420700 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.420703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.421289 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.421290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.421444 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.422038 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:52.422048 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:52.423585 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:52.425315 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.425319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.425321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.425528 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.425796 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.425796 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.425797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.425893 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.426116 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:52.426120 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:52.426233 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.426309 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.426426 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260924 13:37:52.427102 28156 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33565
I20260924 13:37:52.427186 28156 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:52.427379 28156 heartbeater.cc:511] Master 127.25.240.126:33565 requested a full tablet report, sending...
I20260924 13:37:52.427755 27673 ts_manager.cc:194] Registered new tserver with Master: dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67:39935)
I20260924 13:37:52.428242 27673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:40175
I20260924 13:37:52.431787 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 28025
I20260924 13:37:52.431886 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:37:52.432096 26561 external_mini_cluster.cc:1428] Setting key 0bb66ae5f98db7c64b798ba471054a76
I20260924 13:37:52.432911 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:37:52.433776 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.428157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:34833 (local address 127.25.240.126:33565)
0924 13:37:52.428201 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.428203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.428317 (+ 114us) server_negotiation.cc:407] Connection header received
0924 13:37:52.428414 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.428415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.428426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.428497 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:52.428851 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.429288 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.429984 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.430158 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.430269 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.431175 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:52.431176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:52.431176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:52.431181 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:52.432770 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.433117 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.433118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.433118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.433167 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.433347 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.433347 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.433347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.433429 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:52.433473 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.433657 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.433722 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:37:52.433890 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.428093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33565 (local address 127.25.240.67:34833)
0924 13:37:52.428243 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.428259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.428329 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.428502 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.428503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.428723 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:52.428776 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.428778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.429371 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.429372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.429918 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.429920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.430270 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.430270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.430359 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.430830 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:52.430834 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:52.431109 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:52.432843 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.432844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.432844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.433056 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.433230 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:52.433230 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:52.433231 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:52.433293 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:52.433486 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:52.433487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:52.433535 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.433635 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.433742 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
Sep 24 13:37:52 dist-test-slave-m5z1 krb5kdc[27628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:37:52 dist-test-slave-m5z1 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:52.455246 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.448752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.25.240.126:33565)
0924 13:37:52.448807 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.448809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.448832 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:37:52.448880 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.448881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.448889 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.448928 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:52.449438 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.449948 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.450604 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.450772 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.450819 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.452828 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:52.452830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:52.452830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:52.452837 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:52.454128 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.454485 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.454487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.454487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.454550 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:52.454748 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:52.454749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:52.454749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:52.454853 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:52.454959 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.455072 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.455147 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:37:52.456894 27673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:37:52.458454 27673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:52.464715 28175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.460691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:39935 (local address 127.0.0.1:56210)
0924 13:37:52.460850 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.460889 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.460954 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.461544 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.461545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.461546 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:52.461606 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.461607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.462769 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.462770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.463643 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.463645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.464465 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.464466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.464549 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.464559 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.464631 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.464665 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260924 13:37:52.464797 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.461143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.25.240.67:39935)
0924 13:37:52.461319 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.461323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.461333 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:37:52.461366 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.461368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.461383 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.461454 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:52.461683 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.462710 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.463731 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.464369 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.464445 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.464498 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.464662 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.464702 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260924 13:37:52.465348 28174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.460691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:38673 (local address 127.0.0.1:51508)
0924 13:37:52.461056 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.461071 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.461101 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.461696 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.461697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.461699 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:52.461743 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.461744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.462625 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.462626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.463960 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.463962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.465113 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.465114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.465175 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.465179 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.465238 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.465279 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":63,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260924 13:37:52.465438 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.461308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.25.240.66:38673)
0924 13:37:52.461446 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.461450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.461459 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:37:52.461519 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.461521 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.461535 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.461627 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:52.461825 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.462533 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.464049 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.464796 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.464869 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.464924 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.465270 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.465340 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260924 13:37:52.465453 28090 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5528c67d43439dbd1a24137294534c (DEFAULT_TABLE table=test-table [id=2822f6cf602e417591d74e4cbabd04ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:52.465898 28090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5528c67d43439dbd1a24137294534c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:52.466115 28173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.460667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:41237 (local address 127.0.0.1:38900)
0924 13:37:52.460845 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.460889 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.460954 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.461305 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.461308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.461310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:52.461376 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.461378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.462687 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.462687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.463958 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.463960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.465890 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.465891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.465955 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.465959 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.466002 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.466039 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":78,"thread_start_us":56,"threads_started":1}
I20260924 13:37:52.466441 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.460966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.25.240.65:41237)
0924 13:37:52.461125 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.461129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.461140 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:37:52.461182 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.461185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.461202 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.461292 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:52.461468 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.462618 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.464958 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.465673 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.465739 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.466259 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.466284 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.466324 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260924 13:37:52.468501 28179 tablet_bootstrap.cc:492] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Bootstrap starting.
I20260924 13:37:52.468868 27950 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5528c67d43439dbd1a24137294534c (DEFAULT_TABLE table=test-table [id=2822f6cf602e417591d74e4cbabd04ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:52.469183 27950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5528c67d43439dbd1a24137294534c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:52.469529 28179 tablet_bootstrap.cc:654] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:52.469897 28179 log.cc:824] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:52.470016 27810 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5528c67d43439dbd1a24137294534c (DEFAULT_TABLE table=test-table [id=2822f6cf602e417591d74e4cbabd04ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:52.470361 27810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5528c67d43439dbd1a24137294534c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:52.470674 28179 tablet_bootstrap.cc:492] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: No bootstrap required, opened a new log
I20260924 13:37:52.470743 28179 ts_tablet_manager.cc:1397] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:37:52.472211 28181 tablet_bootstrap.cc:492] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Bootstrap starting.
I20260924 13:37:52.472828 28181 tablet_bootstrap.cc:654] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:52.472877 28179 raft_consensus.cc:348] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.473078 28179 raft_consensus.cc:374] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:52.473132 28179 raft_consensus.cc:729] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb3688b8e5d4660b712c0b263973f58, State: Initialized, Role: FOLLOWER
I20260924 13:37:52.473183 28181 log.cc:824] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:52.473269 28179 consensus_queue.cc:260] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.473584 28179 ts_tablet_manager.cc:1428] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 13:37:52.473618 28156 heartbeater.cc:503] Master 127.25.240.126:33565 was elected leader, sending a full tablet report...
I20260924 13:37:52.473949 28184 tablet_bootstrap.cc:492] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Bootstrap starting.
I20260924 13:37:52.474468 28181 tablet_bootstrap.cc:492] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: No bootstrap required, opened a new log
I20260924 13:37:52.474562 28181 ts_tablet_manager.cc:1397] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:37:52.474615 28184 tablet_bootstrap.cc:654] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:52.474931 28184 log.cc:824] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:52.475564 28184 tablet_bootstrap.cc:492] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: No bootstrap required, opened a new log
I20260924 13:37:52.475616 28184 ts_tablet_manager.cc:1397] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:37:52.476008 28181 raft_consensus.cc:348] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.476148 28181 raft_consensus.cc:374] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:52.476192 28181 raft_consensus.cc:729] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cc2d35e9004341b0d7f3079333ce1b, State: Initialized, Role: FOLLOWER
I20260924 13:37:52.476325 28181 consensus_queue.cc:260] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.476583 28181 ts_tablet_manager.cc:1428] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:37:52.476660 28016 heartbeater.cc:503] Master 127.25.240.126:33565 was elected leader, sending a full tablet report...
I20260924 13:37:52.477421 28184 raft_consensus.cc:348] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.477595 28184 raft_consensus.cc:374] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:52.477644 28184 raft_consensus.cc:729] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03b768e9b0e40d3a046a08745e4a22d, State: Initialized, Role: FOLLOWER
I20260924 13:37:52.477753 28184 consensus_queue.cc:260] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.477994 28184 ts_tablet_manager.cc:1428] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 13:37:52.478055 27876 heartbeater.cc:503] Master 127.25.240.126:33565 was elected leader, sending a full tablet report...
W20260924 13:37:52.512269 28017 tablet.cc:2369] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:37:52.604549 27877 tablet.cc:2369] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:37:52.674587 28157 tablet.cc:2369] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:37:52.812043 28183 raft_consensus.cc:482] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:52.812184 28183 raft_consensus.cc:504] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.812515 28186 raft_consensus.cc:482] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:52.812640 28183 leader_election.cc:288] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237), 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673)
I20260924 13:37:52.812623 28186 raft_consensus.cc:504] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.813472 28186 leader_election.cc:288] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237), dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67:39935)
I20260924 13:37:52.816408 28049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.812851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:38673 (local address 127.25.240.67:60915)
0924 13:37:52.812895 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.812908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.812964 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.813181 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.813182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.813184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:52.813260 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.813262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.814235 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.814236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.815460 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.815462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.816155 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.816156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.816236 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.816240 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.816287 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.816323 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":34}
I20260924 13:37:52.816663 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.813022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:60915 (local address 127.25.240.66:38673)
0924 13:37:52.813056 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.813058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.813068 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:37:52.813097 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.813098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.813107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.813163 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:52.813433 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.814156 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.815551 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.816061 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.816118 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.816126 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.816570 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.816605 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":14}
I20260924 13:37:52.816974 28176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.812917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:39155 (local address 127.25.240.65:41237)
0924 13:37:52.812964 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.812967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.813136 (+ 169us) server_negotiation.cc:407] Connection header received
0924 13:37:52.813334 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.813335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.813346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.813379 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:52.813653 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.814332 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.816057 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.816594 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.816634 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.816641 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.816871 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.816912 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:37:52.816988 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.812862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:41237 (local address 127.25.240.67:39155)
0924 13:37:52.813068 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.813087 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.813244 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.813450 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.813451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.813453 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:52.813530 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.813532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.814445 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.814445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.815959 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.815962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.816710 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.816711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.816780 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.816783 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.816856 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.816891 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":124,"threads_started":1}
I20260924 13:37:52.817057 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5528c67d43439dbd1a24137294534c" candidate_uuid: "dfb3688b8e5d4660b712c0b263973f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cc2d35e9004341b0d7f3079333ce1b" is_pre_election: true
I20260924 13:37:52.817160 27970 raft_consensus.cc:2454] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb3688b8e5d4660b712c0b263973f58 in term 0.
I20260924 13:37:52.817346 27830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5528c67d43439dbd1a24137294534c" candidate_uuid: "dfb3688b8e5d4660b712c0b263973f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03b768e9b0e40d3a046a08745e4a22d" is_pre_election: true
I20260924 13:37:52.817394 28044 leader_election.cc:302] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cc2d35e9004341b0d7f3079333ce1b, dfb3688b8e5d4660b712c0b263973f58; no voters:
I20260924 13:37:52.817487 27830 raft_consensus.cc:2454] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb3688b8e5d4660b712c0b263973f58 in term 0.
I20260924 13:37:52.817531 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.813352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:39935 (local address 127.25.240.66:47057)
0924 13:37:52.814044 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.814053 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.814120 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.814264 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.814264 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.814266 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:52.814316 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.814318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.815219 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.815220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.816305 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.816308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.817213 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.817213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.817281 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.817284 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.817415 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.817454 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":52,"threads_started":1}
I20260924 13:37:52.817531 28183 raft_consensus.cc:2781] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:52.817600 28183 raft_consensus.cc:482] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:52.817623 28183 raft_consensus.cc:3037] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:52.817683 28190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.813350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:41237 (local address 127.25.240.66:59793)
0924 13:37:52.813617 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:52.813627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:52.813670 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:52.813978 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:52.813979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:52.813981 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:52.814023 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.814024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.814843 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.814844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.815978 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:52.815980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.817425 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.817426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:52.817490 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.817493 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:37:52.817543 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:37:52.817607 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":55,"threads_started":1}
I20260924 13:37:52.817888 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.813630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:59793 (local address 127.25.240.65:41237)
0924 13:37:52.813802 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.813804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.813811 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:37:52.813834 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.813835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.813843 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.813922 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:52.814085 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.814755 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.816066 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.817000 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.817393 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.817401 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.817771 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.817812 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
I20260924 13:37:52.817902 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.813525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:47057 (local address 127.25.240.67:39935)
0924 13:37:52.813595 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.813597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.814098 (+ 501us) server_negotiation.cc:407] Connection header received
0924 13:37:52.814183 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.814183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.814191 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.814216 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:52.814387 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.815109 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.816390 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.816905 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.816944 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.816950 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.817367 (+ 417us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.817401 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:37:52.818116 28110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5528c67d43439dbd1a24137294534c" candidate_uuid: "96cc2d35e9004341b0d7f3079333ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb3688b8e5d4660b712c0b263973f58" is_pre_election: true
I20260924 13:37:52.818252 27830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5528c67d43439dbd1a24137294534c" candidate_uuid: "96cc2d35e9004341b0d7f3079333ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03b768e9b0e40d3a046a08745e4a22d" is_pre_election: true
I20260924 13:37:52.818322 27830 raft_consensus.cc:2454] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cc2d35e9004341b0d7f3079333ce1b in term 0.
I20260924 13:37:52.818511 27903 leader_election.cc:302] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cc2d35e9004341b0d7f3079333ce1b, d03b768e9b0e40d3a046a08745e4a22d; no voters:
I20260924 13:37:52.818619 28186 raft_consensus.cc:2781] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:52.818598 28183 raft_consensus.cc:504] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.818670 28186 raft_consensus.cc:482] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:52.818722 28186 raft_consensus.cc:3037] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:52.818740 28183 leader_election.cc:288] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [CANDIDATE]: Term 1 election: Requested vote from peers d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237), 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673)
I20260924 13:37:52.818910 27830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5528c67d43439dbd1a24137294534c" candidate_uuid: "dfb3688b8e5d4660b712c0b263973f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03b768e9b0e40d3a046a08745e4a22d"
I20260924 13:37:52.818907 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5528c67d43439dbd1a24137294534c" candidate_uuid: "dfb3688b8e5d4660b712c0b263973f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cc2d35e9004341b0d7f3079333ce1b"
I20260924 13:37:52.818996 27830 raft_consensus.cc:3037] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:52.819237 28110 raft_consensus.cc:2379] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 96cc2d35e9004341b0d7f3079333ce1b in current term 1: Already voted for candidate dfb3688b8e5d4660b712c0b263973f58 in this term.
I20260924 13:37:52.819561 28186 raft_consensus.cc:504] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.819712 28186 leader_election.cc:288] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [CANDIDATE]: Term 1 election: Requested vote from peers d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237), dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67:39935)
I20260924 13:37:52.819806 27830 raft_consensus.cc:2454] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb3688b8e5d4660b712c0b263973f58 in term 1.
I20260924 13:37:52.819892 28110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5528c67d43439dbd1a24137294534c" candidate_uuid: "96cc2d35e9004341b0d7f3079333ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb3688b8e5d4660b712c0b263973f58"
I20260924 13:37:52.819955 28110 raft_consensus.cc:2379] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 96cc2d35e9004341b0d7f3079333ce1b in current term 1: Already voted for candidate dfb3688b8e5d4660b712c0b263973f58 in this term.
I20260924 13:37:52.819970 27970 raft_consensus.cc:2379] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dfb3688b8e5d4660b712c0b263973f58 in current term 1: Already voted for candidate 96cc2d35e9004341b0d7f3079333ce1b in this term.
I20260924 13:37:52.819962 28043 leader_election.cc:302] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d03b768e9b0e40d3a046a08745e4a22d, dfb3688b8e5d4660b712c0b263973f58; no voters:
I20260924 13:37:52.820010 27830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5528c67d43439dbd1a24137294534c" candidate_uuid: "96cc2d35e9004341b0d7f3079333ce1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03b768e9b0e40d3a046a08745e4a22d"
I20260924 13:37:52.820086 27830 raft_consensus.cc:2379] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 96cc2d35e9004341b0d7f3079333ce1b in current term 1: Already voted for candidate dfb3688b8e5d4660b712c0b263973f58 in this term.
I20260924 13:37:52.820163 28183 raft_consensus.cc:2781] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:52.820274 27903 leader_election.cc:302] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96cc2d35e9004341b0d7f3079333ce1b; no voters: d03b768e9b0e40d3a046a08745e4a22d, dfb3688b8e5d4660b712c0b263973f58
I20260924 13:37:52.820291 28183 raft_consensus.cc:686] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 1 LEADER]: Becoming Leader. State: Replica: dfb3688b8e5d4660b712c0b263973f58, State: Running, Role: LEADER
I20260924 13:37:52.820361 28186 raft_consensus.cc:2726] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 13:37:52.820390 28183 consensus_queue.cc:237] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } }
I20260924 13:37:52.821226 27673 catalog_manager.cc:5495] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67). New cstate: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } health_report { overall_health: UNKNOWN } } }
I20260924 13:37:52.907003 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:52.904573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.25.240.67:39935)
0924 13:37:52.904629 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:52.904631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:52.904640 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:37:52.904722 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:52.904722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:52.904739 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:52.904796 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:52.904933 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.905391 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.906217 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:52.906394 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:52.906441 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:52.906648 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:52.906756 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:52.906853 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:52.906896 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 13:37:52.906919 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 13:37:52.908762 27970 raft_consensus.cc:1261] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Refusing update from remote peer dfb3688b8e5d4660b712c0b263973f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:52.908777 27830 raft_consensus.cc:1261] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Refusing update from remote peer dfb3688b8e5d4660b712c0b263973f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:52.909045 28183 consensus_queue.cc:1035] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:52.909175 28192 consensus_queue.cc:1035] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:52.911515 28195 mvcc.cc:205] Tried to move back new op lower bound from 7332892970631573504 to 7332892970272837632. Current Snapshot: MvccSnapshot[applied={T|T < 7332892970631573504}]
I20260924 13:37:52.911785 28196 mvcc.cc:205] Tried to move back new op lower bound from 7332892970631573504 to 7332892970272837632. Current Snapshot: MvccSnapshot[applied={T|T < 7332892970631573504}]
I20260924 13:37:52.912071 28197 mvcc.cc:205] Tried to move back new op lower bound from 7332892970631573504 to 7332892970272837632. Current Snapshot: MvccSnapshot[applied={T|T < 7332892970631573504 or (T in {7332892970631573504})}]
I20260924 13:37:52.915334 27672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:52.915426 27672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:52.916179 27672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa: Sending DeleteTablet for 3 replicas of tablet 4e5528c67d43439dbd1a24137294534c
I20260924 13:37:52.916520 28090 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5528c67d43439dbd1a24137294534c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:52 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:56210
I20260924 13:37:52.916553 27950 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5528c67d43439dbd1a24137294534c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:52 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:51508
I20260924 13:37:52.916752 28206 tablet_replica.cc:330] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: stopping tablet replica
I20260924 13:37:52.916751 27810 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5528c67d43439dbd1a24137294534c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:52 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:38900
I20260924 13:37:52.916858 28206 raft_consensus.cc:2229] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:52.916976 28206 raft_consensus.cc:2258] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:52.916993 28207 tablet_replica.cc:330] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: stopping tablet replica
I20260924 13:37:52.917214 27672 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:37:52.917371 27672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:52.917450 28206 ts_tablet_manager.cc:1905] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:52.917318 28207 raft_consensus.cc:2229] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:52.917714 28207 raft_consensus.cc:2258] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:52.917825 28205 tablet_replica.cc:330] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: stopping tablet replica
I20260924 13:37:52.918001 28205 raft_consensus.cc:2229] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:37:52.918334 28205 raft_consensus.cc:2258] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:52.918354 28207 ts_tablet_manager.cc:1905] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:52.918725 28090 tablet_service.cc:1467] Processing CreateTablet for tablet c03f6f1ff5f945ec93cbabe8930459c7 (DEFAULT_TABLE table=test-table [id=352ec6f57b7e4e5fa138d764fc6d4f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:52.918725 27950 tablet_service.cc:1467] Processing CreateTablet for tablet c03f6f1ff5f945ec93cbabe8930459c7 (DEFAULT_TABLE table=test-table [id=352ec6f57b7e4e5fa138d764fc6d4f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:52.918838 28090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c03f6f1ff5f945ec93cbabe8930459c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:52.918838 27950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c03f6f1ff5f945ec93cbabe8930459c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:52.920094 28179 tablet_bootstrap.cc:492] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: Bootstrap starting.
I20260924 13:37:52.920104 28181 tablet_bootstrap.cc:492] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: Bootstrap starting.
I20260924 13:37:52.920522 28179 tablet_bootstrap.cc:654] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:52.920830 28181 tablet_bootstrap.cc:654] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:52.921075 28207 ts_tablet_manager.cc:1918] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:37:52.921202 27810 tablet_service.cc:1467] Processing CreateTablet for tablet c03f6f1ff5f945ec93cbabe8930459c7 (DEFAULT_TABLE table=test-table [id=352ec6f57b7e4e5fa138d764fc6d4f7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:52.921308 27810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c03f6f1ff5f945ec93cbabe8930459c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:52.921480 28205 ts_tablet_manager.cc:1905] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:52.921885 28206 ts_tablet_manager.cc:1918] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:37:52.921944 28206 log.cc:1196] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/4e5528c67d43439dbd1a24137294534c
I20260924 13:37:52.922531 28206 ts_tablet_manager.cc:1939] T 4e5528c67d43439dbd1a24137294534c P 96cc2d35e9004341b0d7f3079333ce1b: Deleting consensus metadata
I20260924 13:37:52.922919 27660 catalog_manager.cc:4841] TS 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673): tablet 4e5528c67d43439dbd1a24137294534c (table test-table [id=2822f6cf602e417591d74e4cbabd04ab]) successfully deleted
I20260924 13:37:52.923189 28205 ts_tablet_manager.cc:1918] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:37:52.923234 28205 log.cc:1196] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/4e5528c67d43439dbd1a24137294534c
I20260924 13:37:52.923403 28205 ts_tablet_manager.cc:1939] T 4e5528c67d43439dbd1a24137294534c P dfb3688b8e5d4660b712c0b263973f58: Deleting consensus metadata
I20260924 13:37:52.923777 28181 tablet_bootstrap.cc:492] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: No bootstrap required, opened a new log
I20260924 13:37:52.923816 28181 ts_tablet_manager.cc:1397] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 13:37:52.921197 28207 log.cc:1196] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/4e5528c67d43439dbd1a24137294534c
I20260924 13:37:52.923975 27657 catalog_manager.cc:4841] TS dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67:39935): tablet 4e5528c67d43439dbd1a24137294534c (table test-table [id=2822f6cf602e417591d74e4cbabd04ab]) successfully deleted
I20260924 13:37:52.923957 28181 raft_consensus.cc:348] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:52.924046 28181 raft_consensus.cc:374] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:52.924088 28181 raft_consensus.cc:729] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cc2d35e9004341b0d7f3079333ce1b, State: Initialized, Role: FOLLOWER
I20260924 13:37:52.924137 28181 consensus_queue.cc:260] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:52.924253 28181 ts_tablet_manager.cc:1428] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:52.924352 28184 tablet_bootstrap.cc:492] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: Bootstrap starting.
I20260924 13:37:52.924427 28179 tablet_bootstrap.cc:492] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: No bootstrap required, opened a new log
I20260924 13:37:52.924472 28179 ts_tablet_manager.cc:1397] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 13:37:52.924610 28207 ts_tablet_manager.cc:1939] T 4e5528c67d43439dbd1a24137294534c P d03b768e9b0e40d3a046a08745e4a22d: Deleting consensus metadata
I20260924 13:37:52.924647 28179 raft_consensus.cc:348] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:52.924729 28179 raft_consensus.cc:374] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:52.924767 28179 raft_consensus.cc:729] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb3688b8e5d4660b712c0b263973f58, State: Initialized, Role: FOLLOWER
I20260924 13:37:52.924793 28184 tablet_bootstrap.cc:654] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:52.924819 28179 consensus_queue.cc:260] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:52.924934 28179 ts_tablet_manager.cc:1428] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:52.925372 27659 catalog_manager.cc:4841] TS d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237): tablet 4e5528c67d43439dbd1a24137294534c (table test-table [id=2822f6cf602e417591d74e4cbabd04ab]) successfully deleted
I20260924 13:37:52.925824 28184 tablet_bootstrap.cc:492] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: No bootstrap required, opened a new log
I20260924 13:37:52.925873 28184 ts_tablet_manager.cc:1397] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:37:52.926018 28184 raft_consensus.cc:348] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:52.926088 28184 raft_consensus.cc:374] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:52.926103 28184 raft_consensus.cc:729] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03b768e9b0e40d3a046a08745e4a22d, State: Initialized, Role: FOLLOWER
I20260924 13:37:52.926172 28184 consensus_queue.cc:260] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:52.926304 28184 ts_tablet_manager.cc:1428] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:53.001911 28187 raft_consensus.cc:482] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:53.002008 28187 raft_consensus.cc:504] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.002352 28187 leader_election.cc:288] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673), dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67:39935)
I20260924 13:37:53.005755 28178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.002575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:42117 (local address 127.25.240.66:38673)
0924 13:37:53.002622 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.002624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.002699 (+ 75us) server_negotiation.cc:407] Connection header received
0924 13:37:53.002811 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.002812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.002822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.002853 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:53.003107 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.003848 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.004933 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.005314 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.005388 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.005396 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.005628 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.005667 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:37:53.005803 28211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.002508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:38673 (local address 127.25.240.65:42117)
0924 13:37:53.002638 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:53.002648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:53.002729 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:53.002922 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:53.002923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:53.002939 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:53.003012 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.003019 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.003985 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.003986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.004832 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.004834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.005419 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.005421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.005507 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.005537 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 13:37:53.005622 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:37:53.005674 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":25,"thread_start_us":59,"threads_started":1}
I20260924 13:37:53.006103 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c03f6f1ff5f945ec93cbabe8930459c7" candidate_uuid: "d03b768e9b0e40d3a046a08745e4a22d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cc2d35e9004341b0d7f3079333ce1b" is_pre_election: true
I20260924 13:37:53.006198 27970 raft_consensus.cc:2454] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03b768e9b0e40d3a046a08745e4a22d in term 0.
I20260924 13:37:53.006246 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.002714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:44219 (local address 127.25.240.67:39935)
0924 13:37:53.002754 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.002756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.002803 (+ 47us) server_negotiation.cc:407] Connection header received
0924 13:37:53.002869 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.002870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.002878 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.002936 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:53.003140 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.003853 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.005317 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.005870 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.005947 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.005957 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.006150 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.006185 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:37:53.006449 28212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.002559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:39935 (local address 127.25.240.65:44219)
0924 13:37:53.002743 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:53.002753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:53.002815 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:53.002955 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:53.002956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:53.002958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:53.003039 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.003041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.003945 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.003946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.005220 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.005222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.005994 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.005995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.006068 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.006072 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:53.006317 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 13:37:53.006362 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":47,"thread_start_us":98,"threads_started":1}
I20260924 13:37:53.006438 27764 leader_election.cc:302] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cc2d35e9004341b0d7f3079333ce1b, d03b768e9b0e40d3a046a08745e4a22d; no voters:
I20260924 13:37:53.006712 28187 raft_consensus.cc:2781] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:53.006763 28187 raft_consensus.cc:482] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:53.006791 28187 raft_consensus.cc:3037] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:53.006820 28110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c03f6f1ff5f945ec93cbabe8930459c7" candidate_uuid: "d03b768e9b0e40d3a046a08745e4a22d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb3688b8e5d4660b712c0b263973f58" is_pre_election: true
I20260924 13:37:53.006915 28110 raft_consensus.cc:2454] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d03b768e9b0e40d3a046a08745e4a22d in term 0.
I20260924 13:37:53.007601 28187 raft_consensus.cc:504] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.007746 28187 leader_election.cc:288] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [CANDIDATE]: Term 1 election: Requested vote from peers 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673), dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67:39935)
I20260924 13:37:53.007917 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c03f6f1ff5f945ec93cbabe8930459c7" candidate_uuid: "d03b768e9b0e40d3a046a08745e4a22d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cc2d35e9004341b0d7f3079333ce1b"
I20260924 13:37:53.007921 28110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c03f6f1ff5f945ec93cbabe8930459c7" candidate_uuid: "d03b768e9b0e40d3a046a08745e4a22d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfb3688b8e5d4660b712c0b263973f58"
I20260924 13:37:53.007990 28110 raft_consensus.cc:3037] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:53.007993 27970 raft_consensus.cc:3037] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:53.008589 27970 raft_consensus.cc:2454] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03b768e9b0e40d3a046a08745e4a22d in term 1.
I20260924 13:37:53.008749 27764 leader_election.cc:302] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cc2d35e9004341b0d7f3079333ce1b, d03b768e9b0e40d3a046a08745e4a22d; no voters:
I20260924 13:37:53.008852 28187 raft_consensus.cc:2781] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:53.008895 28110 raft_consensus.cc:2454] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d03b768e9b0e40d3a046a08745e4a22d in term 1.
I20260924 13:37:53.009004 28187 raft_consensus.cc:686] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 1 LEADER]: Becoming Leader. State: Replica: d03b768e9b0e40d3a046a08745e4a22d, State: Running, Role: LEADER
I20260924 13:37:53.009093 28187 consensus_queue.cc:237] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.009825 27672 catalog_manager.cc:5495] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d reported cstate change: term changed from 0 to 1, leader changed from <none> to d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65). New cstate: current_term: 1 leader_uuid: "d03b768e9b0e40d3a046a08745e4a22d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } health_report { overall_health: UNKNOWN } } }
Sep 24 13:37:53 dist-test-slave-m5z1 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257071, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:53.040995 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.034230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33565 (local address 127.0.0.1:47128)
0924 13:37:53.034424 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:53.034436 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:53.034535 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:53.034719 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:53.034720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:53.035060 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:53.035118 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.035120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.035849 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.035850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.036418 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.036419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.036738 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.036739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.036778 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.037323 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:53.037327 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:53.038719 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:53.040095 (+ 1376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:53.040102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:53.040104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:53.040295 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:53.040470 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:53.040471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:53.040471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:53.040525 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:53.040722 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:53.040728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:53.040767 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 13:37:53.040817 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:37:53.040904 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260924 13:37:53.041091 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.034293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47128 (local address 127.25.240.126:33565)
0924 13:37:53.034434 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.034437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.034514 (+ 77us) server_negotiation.cc:407] Connection header received
0924 13:37:53.034614 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.034615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.034626 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.034698 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:53.035239 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.035773 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.036483 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.036676 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.036724 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.038796 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:53.038797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:53.038797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:53.038803 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:53.040021 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:53.040359 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:53.040359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:53.040360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:53.040413 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:53.040575 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:53.040575 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:53.040576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:53.040669 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:53.040717 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.040846 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.040952 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260924 13:37:53.042868 27672 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:47128:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 13:37:53.043048 27672 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:53.044457 27810 tablet_service.cc:1467] Processing CreateTablet for tablet b330125fa6544a958d163e811d17608f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc2f774fae2942f9a004ed1c9f90e160]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:37:53.044487 28090 tablet_service.cc:1467] Processing CreateTablet for tablet b330125fa6544a958d163e811d17608f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc2f774fae2942f9a004ed1c9f90e160]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:37:53.044623 27810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b330125fa6544a958d163e811d17608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:53.044647 28090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b330125fa6544a958d163e811d17608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:53.045804 28184 tablet_bootstrap.cc:492] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: Bootstrap starting.
I20260924 13:37:53.045845 28179 tablet_bootstrap.cc:492] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: Bootstrap starting.
I20260924 13:37:53.046186 28179 tablet_bootstrap.cc:654] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:53.046204 28184 tablet_bootstrap.cc:654] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:53.046551 27950 tablet_service.cc:1467] Processing CreateTablet for tablet b330125fa6544a958d163e811d17608f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dc2f774fae2942f9a004ed1c9f90e160]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:37:53.046685 27950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b330125fa6544a958d163e811d17608f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:53.046808 28179 tablet_bootstrap.cc:492] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: No bootstrap required, opened a new log
I20260924 13:37:53.046844 28184 tablet_bootstrap.cc:492] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: No bootstrap required, opened a new log
I20260924 13:37:53.046844 28179 ts_tablet_manager.cc:1397] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:53.046878 28184 ts_tablet_manager.cc:1397] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:37:53.047021 28184 raft_consensus.cc:348] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.047093 28184 raft_consensus.cc:374] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:53.047114 28184 raft_consensus.cc:729] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d03b768e9b0e40d3a046a08745e4a22d, State: Initialized, Role: FOLLOWER
I20260924 13:37:53.047065 28179 raft_consensus.cc:348] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.047155 28179 raft_consensus.cc:374] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:53.047178 28179 raft_consensus.cc:729] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfb3688b8e5d4660b712c0b263973f58, State: Initialized, Role: FOLLOWER
I20260924 13:37:53.047164 28184 consensus_queue.cc:260] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.047226 28179 consensus_queue.cc:260] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.047254 28184 ts_tablet_manager.cc:1428] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:53.047307 28179 ts_tablet_manager.cc:1428] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:53.047340 28200 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.047420 28200 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:53.048429 28181 tablet_bootstrap.cc:492] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: Bootstrap starting.
I20260924 13:37:53.048520 28213 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.048612 28213 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:53.048899 28181 tablet_bootstrap.cc:654] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:53.049463 28181 tablet_bootstrap.cc:492] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: No bootstrap required, opened a new log
I20260924 13:37:53.049526 28181 ts_tablet_manager.cc:1397] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:53.049690 28181 raft_consensus.cc:348] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.049770 28181 raft_consensus.cc:374] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:53.049794 28181 raft_consensus.cc:729] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cc2d35e9004341b0d7f3079333ce1b, State: Initialized, Role: FOLLOWER
I20260924 13:37:53.049845 28181 consensus_queue.cc:260] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.049932 28181 ts_tablet_manager.cc:1428] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:53.049997 28186 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.050109 28186 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:53.186600 28200 raft_consensus.cc:482] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:53.186719 28200 raft_consensus.cc:504] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.186900 28200 leader_election.cc:288] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237), 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673)
I20260924 13:37:53.187179 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b330125fa6544a958d163e811d17608f" candidate_uuid: "dfb3688b8e5d4660b712c0b263973f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cc2d35e9004341b0d7f3079333ce1b" is_pre_election: true
I20260924 13:37:53.187175 27830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b330125fa6544a958d163e811d17608f" candidate_uuid: "dfb3688b8e5d4660b712c0b263973f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03b768e9b0e40d3a046a08745e4a22d" is_pre_election: true
I20260924 13:37:53.187283 27970 raft_consensus.cc:2454] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb3688b8e5d4660b712c0b263973f58 in term 0.
I20260924 13:37:53.187287 27830 raft_consensus.cc:2454] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfb3688b8e5d4660b712c0b263973f58 in term 0.
I20260924 13:37:53.187449 28044 leader_election.cc:302] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cc2d35e9004341b0d7f3079333ce1b, dfb3688b8e5d4660b712c0b263973f58; no voters:
I20260924 13:37:53.187561 28200 raft_consensus.cc:2781] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:53.187602 28200 raft_consensus.cc:482] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:53.187621 28200 raft_consensus.cc:3037] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:53.188340 28200 raft_consensus.cc:504] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.188493 28200 leader_election.cc:288] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [CANDIDATE]: Term 1 election: Requested vote from peers d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237), 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673)
I20260924 13:37:53.188616 27830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b330125fa6544a958d163e811d17608f" candidate_uuid: "dfb3688b8e5d4660b712c0b263973f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d03b768e9b0e40d3a046a08745e4a22d"
I20260924 13:37:53.188694 27830 raft_consensus.cc:3037] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:53.188664 27970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b330125fa6544a958d163e811d17608f" candidate_uuid: "dfb3688b8e5d4660b712c0b263973f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cc2d35e9004341b0d7f3079333ce1b"
I20260924 13:37:53.188735 27970 raft_consensus.cc:3037] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:53.189420 27830 raft_consensus.cc:2454] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb3688b8e5d4660b712c0b263973f58 in term 1.
I20260924 13:37:53.189424 27970 raft_consensus.cc:2454] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfb3688b8e5d4660b712c0b263973f58 in term 1.
I20260924 13:37:53.189610 28043 leader_election.cc:302] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d03b768e9b0e40d3a046a08745e4a22d, dfb3688b8e5d4660b712c0b263973f58; no voters:
I20260924 13:37:53.189716 28200 raft_consensus.cc:2781] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:53.189795 28200 raft_consensus.cc:686] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [term 1 LEADER]: Becoming Leader. State: Replica: dfb3688b8e5d4660b712c0b263973f58, State: Running, Role: LEADER
I20260924 13:37:53.189853 28200 consensus_queue.cc:237] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } }
I20260924 13:37:53.190109 28192 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: TxnStatusTablet state changed. Reason: New leader dfb3688b8e5d4660b712c0b263973f58. Latest consensus state: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.190200 28192 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:37:53.190359 28226 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 13:37:53.190496 28226 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 13:37:53.190436 27673 catalog_manager.cc:5495] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67). New cstate: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } health_report { overall_health: HEALTHY } } }
I20260924 13:37:53.229853 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.227492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.25.240.67:39935)
0924 13:37:53.227561 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.227564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.227571 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:37:53.227595 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.227596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.227605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.227670 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:53.227841 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.228318 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.229097 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.229304 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.229389 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.229576 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:53.229610 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:53.229685 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.229759 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.229777 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 13:37:53.229900 28220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.227271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:39935 (local address 127.0.0.1:56226)
0924 13:37:53.227359 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:53.227370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:53.227417 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:53.227698 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:53.227699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:53.227700 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:53.227772 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.227774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.228402 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.228403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.228987 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.228989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.229391 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.229392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.229454 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.229469 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:53.229689 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:53.229699 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 13:37:53.229753 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 13:37:53.229802 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 13:37:53.230981 27830 raft_consensus.cc:1261] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Refusing update from remote peer dfb3688b8e5d4660b712c0b263973f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:53.231040 27970 raft_consensus.cc:1261] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Refusing update from remote peer dfb3688b8e5d4660b712c0b263973f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:53.231228 28200 consensus_queue.cc:1035] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:53.231326 28192 consensus_queue.cc:1035] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:53.231851 28213 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: TxnStatusTablet state changed. Reason: New leader dfb3688b8e5d4660b712c0b263973f58. Latest consensus state: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.231943 28213 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:53.232542 28186 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: TxnStatusTablet state changed. Reason: New leader dfb3688b8e5d4660b712c0b263973f58. Latest consensus state: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.232631 28186 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:53.232676 28183 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.232750 28183 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:37:53.233359 28183 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.233430 28183 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P dfb3688b8e5d4660b712c0b263973f58: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:37:53.233690 28186 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.233765 28186 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P 96cc2d35e9004341b0d7f3079333ce1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:53.233956 28213 tablet_replica.cc:439] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dfb3688b8e5d4660b712c0b263973f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d03b768e9b0e40d3a046a08745e4a22d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 41237 } } peers { permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 } } peers { permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 } } }
I20260924 13:37:53.234033 28213 tablet_replica.cc:442] T b330125fa6544a958d163e811d17608f P d03b768e9b0e40d3a046a08745e4a22d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:53.238107 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.235562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.25.240.65:41237)
0924 13:37:53.235608 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.235610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.235640 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:37:53.235713 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.235714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.235734 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.235815 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:53.235891 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.236608 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.237287 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.237610 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.237680 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.237793 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:53.237921 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:53.237989 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.238011 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.238032 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:37:53.242957 28177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.240429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:53815 (local address 127.25.240.67:39935)
0924 13:37:53.240469 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.240471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.240586 (+ 115us) server_negotiation.cc:407] Connection header received
0924 13:37:53.240641 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.240642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.240649 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.240693 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:53.240810 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.241235 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.242293 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.242462 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.242507 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.242699 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:53.242722 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:53.242759 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.242846 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.242861 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:37:53.242970 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.240371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:39935 (local address 127.25.240.65:53815)
0924 13:37:53.240518 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:53.240528 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:53.240587 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:53.240696 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:53.240697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:53.240699 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:53.240759 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.240760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.241330 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.241331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.242209 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.242212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.242532 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.242533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.242609 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.242623 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:53.242771 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:53.242773 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:37:53.242841 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 13:37:53.242880 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 13:37:53.245512 28110 raft_consensus.cc:1261] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Refusing update from remote peer d03b768e9b0e40d3a046a08745e4a22d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:53.245501 27970 raft_consensus.cc:1261] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Refusing update from remote peer d03b768e9b0e40d3a046a08745e4a22d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:53.245779 28213 consensus_queue.cc:1035] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfb3688b8e5d4660b712c0b263973f58" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 39935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:53.245904 28187 consensus_queue.cc:1035] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cc2d35e9004341b0d7f3079333ce1b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 38673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:53.254684 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.251818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:53309 (local address 127.25.240.65:41237)
0924 13:37:53.251858 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.251859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.251872 (+ 13us) server_negotiation.cc:407] Connection header received
0924 13:37:53.251966 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.251967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.251977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.252043 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:53.252177 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.252897 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.254042 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.254267 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.254319 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.254496 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:53.254520 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:53.254567 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.254622 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.254638 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:37:53.254750 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.251676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:41237 (local address 127.25.240.67:53309)
0924 13:37:53.251825 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:53.251837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:53.251883 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:53.252034 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:53.252035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:53.252036 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:53.252089 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.252090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.252999 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.253000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.253972 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:53.253975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.254340 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.254341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:53.254399 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.254414 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:53.254568 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:53.254570 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:37:53.254620 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:37:53.254661 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260924 13:37:53.265156 27672 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:53.265271 27672 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:53.266021 27672 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c5cbb650ad54659bb67d68e1dfc94fa: Sending DeleteTablet for 3 replicas of tablet c03f6f1ff5f945ec93cbabe8930459c7
I20260924 13:37:53.266357 27950 tablet_service.cc:1514] Processing DeleteTablet for tablet c03f6f1ff5f945ec93cbabe8930459c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:53 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:51508
I20260924 13:37:53.266438 27810 tablet_service.cc:1514] Processing DeleteTablet for tablet c03f6f1ff5f945ec93cbabe8930459c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:53 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:38900
I20260924 13:37:53.266470 28206 tablet_replica.cc:330] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: stopping tablet replica
I20260924 13:37:53.266534 28206 raft_consensus.cc:2229] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:53.266543 28207 tablet_replica.cc:330] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: stopping tablet replica
I20260924 13:37:53.266597 28206 raft_consensus.cc:2258] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:53.266597 28207 raft_consensus.cc:2229] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:37:53.266687 28207 raft_consensus.cc:2258] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:53.267467 28206 ts_tablet_manager.cc:1905] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:53.269240 28206 ts_tablet_manager.cc:1918] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:37:53.269296 28206 log.cc:1196] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/c03f6f1ff5f945ec93cbabe8930459c7
I20260924 13:37:53.269297 28207 ts_tablet_manager.cc:1905] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:53.269520 28206 ts_tablet_manager.cc:1939] T c03f6f1ff5f945ec93cbabe8930459c7 P 96cc2d35e9004341b0d7f3079333ce1b: Deleting consensus metadata
I20260924 13:37:53.269958 27660 catalog_manager.cc:4841] TS 96cc2d35e9004341b0d7f3079333ce1b (127.25.240.66:38673): tablet c03f6f1ff5f945ec93cbabe8930459c7 (table test-table [id=352ec6f57b7e4e5fa138d764fc6d4f7a]) successfully deleted
I20260924 13:37:53.271311 28090 tablet_service.cc:1514] Processing DeleteTablet for tablet c03f6f1ff5f945ec93cbabe8930459c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:53 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:56210
I20260924 13:37:53.271416 28205 tablet_replica.cc:330] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: stopping tablet replica
I20260924 13:37:53.271471 28205 raft_consensus.cc:2229] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:53.271516 28205 raft_consensus.cc:2258] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:53.271727 28207 ts_tablet_manager.cc:1918] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:37:53.271788 28207 log.cc:1196] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/c03f6f1ff5f945ec93cbabe8930459c7
I20260924 13:37:53.272015 28207 ts_tablet_manager.cc:1939] T c03f6f1ff5f945ec93cbabe8930459c7 P d03b768e9b0e40d3a046a08745e4a22d: Deleting consensus metadata
I20260924 13:37:53.272333 27659 catalog_manager.cc:4841] TS d03b768e9b0e40d3a046a08745e4a22d (127.25.240.65:41237): tablet c03f6f1ff5f945ec93cbabe8930459c7 (table test-table [id=352ec6f57b7e4e5fa138d764fc6d4f7a]) successfully deleted
I20260924 13:37:53.272626 28205 ts_tablet_manager.cc:1905] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 13:37:53 dist-test-slave-m5z1 krb5kdc[27628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:37:53.274333 28205 ts_tablet_manager.cc:1918] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:37:53.274407 28205 log.cc:1196] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/c03f6f1ff5f945ec93cbabe8930459c7
I20260924 13:37:53.274962 28205 ts_tablet_manager.cc:1939] T c03f6f1ff5f945ec93cbabe8930459c7 P dfb3688b8e5d4660b712c0b263973f58: Deleting consensus metadata
I20260924 13:37:53.275570 27657 catalog_manager.cc:4841] TS dfb3688b8e5d4660b712c0b263973f58 (127.25.240.67:39935): tablet c03f6f1ff5f945ec93cbabe8930459c7 (table test-table [id=352ec6f57b7e4e5fa138d764fc6d4f7a]) successfully deleted
I20260924 13:37:53.277676 28191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.267202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.25.240.65:41237)
0924 13:37:53.267239 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.267241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.267481 (+ 240us) server_negotiation.cc:407] Connection header received
0924 13:37:53.267591 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.267592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.267602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.267632 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:53.268370 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.269101 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.270212 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.270484 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.270523 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.274279 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:53.274286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:53.274286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:53.274293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:53.276275 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:53.276680 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:53.276684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:53.276684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:53.276753 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:53.276946 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:53.276947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:53.276947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:53.277101 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:53.277217 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.277430 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.277511 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:37:53.278126 27870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38912
I20260924 13:37:53.289318 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:53.279043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.25.240.126:33565)
0924 13:37:53.279086 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:53.279088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:53.281743 (+ 2655us) server_negotiation.cc:407] Connection header received
0924 13:37:53.281797 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:53.281798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:53.281812 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:53.281874 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:53.282513 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.283318 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.284357 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:53.284680 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:53.284752 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:53.286316 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:53.286318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:53.286318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:53.286325 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:53.288072 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:53.288447 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:53.288449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:53.288449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:53.288512 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:53.288743 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:53.288744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:53.288744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:53.288879 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:53.288985 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:53.289124 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 13:37:53.289219 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 13:37:53.289516 27672 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47132
I20260924 13:37:53.290154 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 27745
I20260924 13:37:53.295459 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 27885
I20260924 13:37:53.300365 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 28025
I20260924 13:37:53.306149 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 27641
2026-09-24T13:37:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (6457 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:37:53 dist-test-slave-m5z1 krb5kdc[28247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:37:53 dist-test-slave-m5z1 krb5kdc[28247](info): set up 2 sockets
Sep 24 13:37:53 dist-test-slave-m5z1 krb5kdc[28247](info): commencing operation
krb5kdc: starting...
W20260924 13:37:54.357436 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:37:54 dist-test-slave-m5z1 krb5kdc[28247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:37:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:37:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:37:54.476969 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35715
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:35715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:54.556427 28260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:54.556665 28260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:54.556700 28260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:54.557894 28260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:37:54.557953 28260 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 13:37:54.557983 28260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:54.558013 28260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:54.558044 28260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:37:54.558068 28260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:37:54.559687 28260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:35715
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:54.559965 28260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:54.560232 28260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:54.565011 28266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:54.565137 28260 server_base.cc:1034] running on GCE node
W20260924 13:37:54.565004 28268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:54.565025 28265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:54.565594 28260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:54.565857 28260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:54.567126 28260 hybrid_clock.cc:648] HybridClock initialized: now 1790257074567112 us; error 29 us; skew 500 ppm
Sep 24 13:37:54 dist-test-slave-m5z1 krb5kdc[28247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257074, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:54.569725 28260 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:37:54.570184 28260 webserver.cc:460] Webserver started at http://127.25.240.126:40137/ using document root <none> and password file <none>
I20260924 13:37:54.570401 28260 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:54.570446 28260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:54.570556 28260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:54.571578 28260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "c3982224b8494e4287051c2f5b1c97e5"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "8b367a8fa8f79852f771c7979202bcfd"
server_key_iv: "c012a9fd53863eadbcfd89da7978fe2c"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.571936 28260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "c3982224b8494e4287051c2f5b1c97e5"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "8b367a8fa8f79852f771c7979202bcfd"
server_key_iv: "c012a9fd53863eadbcfd89da7978fe2c"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.573240 28260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:37:54.574035 28275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:54.574218 28260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:54.574306 28260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "c3982224b8494e4287051c2f5b1c97e5"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "8b367a8fa8f79852f771c7979202bcfd"
server_key_iv: "c012a9fd53863eadbcfd89da7978fe2c"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.574376 28260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:54.597319 28260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:54.599229 28260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:54.599380 28260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:54.604085 28260 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:35715
I20260924 13:37:54.604157 28337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:35715 every 8 connection(s)
I20260924 13:37:54.604485 28260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:37:54.605101 28338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:54.607365 28338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: Bootstrap starting.
I20260924 13:37:54.607895 28338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:54.608127 28338 log.cc:824] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:54.608743 28338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: No bootstrap required, opened a new log
I20260924 13:37:54.610828 28338 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3982224b8494e4287051c2f5b1c97e5" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35715 } }
I20260924 13:37:54.610963 28338 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:54.610985 28338 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3982224b8494e4287051c2f5b1c97e5, State: Initialized, Role: FOLLOWER
I20260924 13:37:54.611083 28338 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3982224b8494e4287051c2f5b1c97e5" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35715 } }
I20260924 13:37:54.611167 28338 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:37:54.611205 28338 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:37:54.611255 28338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:54.611542 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 28260
I20260924 13:37:54.611614 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:37:54.611773 26561 external_mini_cluster.cc:1428] Setting key a11c50a582ddb278dd5bedbdb82896d7
I20260924 13:37:54.611909 28338 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3982224b8494e4287051c2f5b1c97e5" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35715 } }
I20260924 13:37:54.612022 28338 leader_election.cc:302] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3982224b8494e4287051c2f5b1c97e5; no voters:
I20260924 13:37:54.612187 28338 leader_election.cc:288] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:37:54.612238 28341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:54.612393 28341 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [term 1 LEADER]: Becoming Leader. State: Replica: c3982224b8494e4287051c2f5b1c97e5, State: Running, Role: LEADER
I20260924 13:37:54.612452 28338 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:37:54.612499 28341 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3982224b8494e4287051c2f5b1c97e5" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35715 } }
I20260924 13:37:54.612836 28341 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3982224b8494e4287051c2f5b1c97e5. Latest consensus state: current_term: 1 leader_uuid: "c3982224b8494e4287051c2f5b1c97e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3982224b8494e4287051c2f5b1c97e5" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35715 } } }
I20260924 13:37:54.612847 28343 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3982224b8494e4287051c2f5b1c97e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3982224b8494e4287051c2f5b1c97e5" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35715 } } }
I20260924 13:37:54.612932 28343 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:37:54.612928 28341 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5 [sys.catalog]: This master's current role is: LEADER
I20260924 13:37:54.613627 28350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:37:54.614135 28350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:37:54.615619 28350 catalog_manager.cc:1347] Generated new cluster ID: 4b80fe64ad9c478ca35170d0a2bf64b9
I20260924 13:37:54.615671 28350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:37:54 dist-test-slave-m5z1 krb5kdc[28247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:54.623847 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.612250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.25.240.126:35715)
0924 13:37:54.612789 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:54.612798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:54.613085 (+ 287us) server_negotiation.cc:407] Connection header received
0924 13:37:54.613220 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:54.613226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:54.613274 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:54.613411 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:54.614331 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.615782 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.616821 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.617458 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.617595 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.620283 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:54.620286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:54.620287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:54.620298 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:54.622250 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.622652 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.622655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.622655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.622765 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.622980 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.622980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.622981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.623256 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:54.623400 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:54.623523 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 13:37:54.623654 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":63,"threads_started":1}
I20260924 13:37:54.637250 28350 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:37:54.638028 28350 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:37:54.645503 28350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: Generated new TSK 0
I20260924 13:37:54.645721 28350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:37:54.676457 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35715
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:54.760058 28364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:54.760253 28364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:54.760277 28364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:54.761405 28364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:54.761447 28364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:54.761467 28364 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 13:37:54.761487 28364 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:37:54.761529 28364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:37:54.762998 28364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:54.763257 28364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:54.763497 28364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:54.768404 28369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:54.768404 28372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:54.768460 28370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:54.768635 28364 server_base.cc:1034] running on GCE node
I20260924 13:37:54.768821 28364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:54.769057 28364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:54.770222 28364 hybrid_clock.cc:648] HybridClock initialized: now 1790257074770210 us; error 30 us; skew 500 ppm
Sep 24 13:37:54 dist-test-slave-m5z1 krb5kdc[28247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257074, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:54.772764 28364 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:37:54.773207 28364 webserver.cc:460] Webserver started at http://127.25.240.65:38311/ using document root <none> and password file <none>
I20260924 13:37:54.773448 28364 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:54.773516 28364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:54.773686 28364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:54.774726 28364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "a1e6064e9a714cdbb5907457ebe6cab4"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "ef275bedabd63d5a238313ae3b1e291e"
server_key_iv: "409fa7cb2169055d19561493795c838e"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.775095 28364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "a1e6064e9a714cdbb5907457ebe6cab4"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "ef275bedabd63d5a238313ae3b1e291e"
server_key_iv: "409fa7cb2169055d19561493795c838e"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.776403 28364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:37:54.777390 28379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:54.777592 28364 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:37:54.777693 28364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "a1e6064e9a714cdbb5907457ebe6cab4"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "ef275bedabd63d5a238313ae3b1e291e"
server_key_iv: "409fa7cb2169055d19561493795c838e"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.777781 28364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:54.784673 28364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:54.786456 28364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:54.786615 28364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:54.787076 28364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:54.787141 28364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:54.787185 28364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:54.787210 28364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:54.794420 28364 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:32893
I20260924 13:37:54.794489 28494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:32893 every 8 connection(s)
Sep 24 13:37:54 dist-test-slave-m5z1 krb5kdc[28247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257074, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:54.794787 28364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:37:54.798089 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.787604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:45325 (local address 127.25.240.126:35715)
0924 13:37:54.787660 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:54.787662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:54.787866 (+ 204us) server_negotiation.cc:407] Connection header received
0924 13:37:54.788413 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:54.788415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:54.788432 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:54.788494 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:54.789465 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.790354 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.792089 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.792445 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.792506 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.795277 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:54.795279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:54.795279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:54.795288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:54.796505 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.797084 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.797085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.797086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.797153 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.797423 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.797424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.797424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.797543 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:54.797644 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:54.797915 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 13:37:54.797999 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 13:37:54.798276 28388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.787580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35715 (local address 127.25.240.65:45325)
0924 13:37:54.787753 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:54.787787 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:54.788289 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:54.788609 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:54.788613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:54.789032 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:54.789348 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:54.789356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.790451 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.790452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:54.791967 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:54.791970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.792540 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.792540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:54.792711 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.793327 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:54.793338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:54.795178 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:54.796634 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:54.796641 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:54.796644 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:54.796953 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:54.797224 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:54.797225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:54.797225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:54.797336 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:54.797633 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:54.797640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:54.797779 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 13:37:54.797888 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 13:37:54.798042 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260924 13:37:54.798823 28495 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35715
I20260924 13:37:54.798902 28495 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:54.799078 28495 heartbeater.cc:511] Master 127.25.240.126:35715 requested a full tablet report, sending...
I20260924 13:37:54.799559 28292 ts_manager.cc:194] Registered new tserver with Master: a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893)
I20260924 13:37:54.800256 28292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:45325
I20260924 13:37:54.801270 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 28364
I20260924 13:37:54.801370 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:37:54.801532 26561 external_mini_cluster.cc:1428] Setting key c50d71c781fc177009a9398411340334
I20260924 13:37:54.805971 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.799918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:52397 (local address 127.25.240.126:35715)
0924 13:37:54.799960 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:54.799962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:54.800083 (+ 121us) server_negotiation.cc:407] Connection header received
0924 13:37:54.800153 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:54.800154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:54.800164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:54.800207 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:54.800567 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.801237 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.802201 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.802406 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.802490 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.803400 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:54.803401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:54.803401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:54.803409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:54.804771 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.805166 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.805167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.805167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.805249 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.805480 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.805480 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.805481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.805629 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:54.805689 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:54.805854 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 13:37:54.805916 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:37:54.806092 28501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.799855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35715 (local address 127.25.240.65:52397)
0924 13:37:54.800011 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:54.800027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:54.800096 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:54.800221 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:54.800221 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:54.800435 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:54.800494 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:54.800496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.801331 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.801333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:54.802111 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:54.802114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.802492 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.802493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:54.802565 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.802975 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:54.802979 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:54.803297 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:54.804855 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:54.804856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:54.804856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:54.805090 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:54.805328 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:54.805329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:54.805330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:54.805415 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:54.805698 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:54.805699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:54.805749 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 13:37:54.805829 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:37:54.805935 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:37:54.826514 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35715
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:54.906220 28504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:54.906457 28504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:54.906497 28504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:54.907650 28504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:54.907722 28504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:54.907757 28504 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 13:37:54.907787 28504 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:37:54.907821 28504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:37:54.909255 28504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:54.909528 28504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:54.909773 28504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:54.914777 28509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:54.914788 28510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:54.914903 28512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:54.915021 28504 server_base.cc:1034] running on GCE node
I20260924 13:37:54.915189 28504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:54.915385 28504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:54.916563 28504 hybrid_clock.cc:648] HybridClock initialized: now 1790257074916531 us; error 34 us; skew 500 ppm
Sep 24 13:37:54 dist-test-slave-m5z1 krb5kdc[28247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257074, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:54.919013 28504 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:37:54.919471 28504 webserver.cc:460] Webserver started at http://127.25.240.66:42865/ using document root <none> and password file <none>
I20260924 13:37:54.919703 28504 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:54.919771 28504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:54.919946 28504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:54.921020 28504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "7219f307dc554d5e8b508bb50c7a8ed7"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "235efdc0be2607554ce295c5feba6888"
server_key_iv: "45f4a459cfd593a370c52065534aad60"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.921373 28504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "7219f307dc554d5e8b508bb50c7a8ed7"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "235efdc0be2607554ce295c5feba6888"
server_key_iv: "45f4a459cfd593a370c52065534aad60"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.922709 28504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:37:54.923399 28519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:54.923563 28504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:54.923660 28504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "7219f307dc554d5e8b508bb50c7a8ed7"
format_stamp: "Formatted at 2026-09-24 13:37:54 on dist-test-slave-m5z1"
server_key: "235efdc0be2607554ce295c5feba6888"
server_key_iv: "45f4a459cfd593a370c52065534aad60"
server_key_version: "encryptionkey@0"
I20260924 13:37:54.923738 28504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:54.936800 28504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:54.939098 28504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:54.939260 28504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:54.939782 28504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:54.939841 28504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:54.939881 28504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:54.939910 28504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:37:54 dist-test-slave-m5z1 krb5kdc[28247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257074, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:54.947373 28504 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:41795
I20260924 13:37:54.947494 28634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:41795 every 8 connection(s)
I20260924 13:37:54.947829 28504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:37:54.949947 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.940193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:39889 (local address 127.25.240.126:35715)
0924 13:37:54.940265 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:54.940268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:54.940508 (+ 240us) server_negotiation.cc:407] Connection header received
0924 13:37:54.941209 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:54.941211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:54.941222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:54.941266 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:54.942197 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.942731 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.944418 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.944646 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.944733 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.947086 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:54.947088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:54.947088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:54.947096 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:54.948715 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.949071 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.949072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.949072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.949126 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.949341 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.949341 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.949341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.949459 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:54.949512 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:54.949819 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 13:37:54.949887 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:37:54.950088 28528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.940162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35715 (local address 127.25.240.66:39889)
0924 13:37:54.940355 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:54.940404 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:54.941103 (+ 699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:54.941346 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:54.941351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:54.941769 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:54.942080 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:54.942086 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.942830 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.942831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:54.944316 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:54.944320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.944762 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.944764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:54.944883 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.945397 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:54.945406 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:54.946969 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:54.948785 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:54.948789 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:54.948791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:54.948997 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:54.949188 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:54.949188 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:54.949189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:54.949283 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:54.949517 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:54.949521 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:54.949701 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 13:37:54.949786 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:37:54.949903 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":101,"threads_started":1}
I20260924 13:37:54.950574 28635 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35715
I20260924 13:37:54.950650 28635 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:54.950855 28635 heartbeater.cc:511] Master 127.25.240.126:35715 requested a full tablet report, sending...
I20260924 13:37:54.951050 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 28504
I20260924 13:37:54.951151 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:37:54.951227 28291 ts_manager.cc:194] Registered new tserver with Master: 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795)
I20260924 13:37:54.951332 26561 external_mini_cluster.cc:1428] Setting key 0974d7ea940c2d7f66c8bfefd49042a2
I20260924 13:37:54.951735 28291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:39889
I20260924 13:37:54.958115 28641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.951628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35715 (local address 127.25.240.66:43809)
0924 13:37:54.951746 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:54.951758 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:54.951916 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:54.952125 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:54.952126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:54.952310 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:54.952392 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:54.952394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.953341 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.953342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:54.953947 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:54.953949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.954411 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.954412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:54.954471 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.954891 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:54.954895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:54.955163 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:54.957071 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:54.957072 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:54.957072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:54.957240 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:54.957456 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:54.957457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:54.957457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:54.957507 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:54.957869 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:54.957870 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:54.957906 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 13:37:54.957962 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:37:54.958035 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":76,"threads_started":1}
I20260924 13:37:54.958189 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:54.951639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:43809 (local address 127.25.240.126:35715)
0924 13:37:54.951682 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:54.951685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:54.951809 (+ 124us) server_negotiation.cc:407] Connection header received
0924 13:37:54.952014 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:54.952015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:54.952025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:54.952054 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:54.952481 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.953213 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.954033 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:54.954336 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:54.954410 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:54.955252 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:54.955253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:54.955253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:54.955260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:54.956980 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.957312 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.957313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.957314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.957375 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:54.957591 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:54.957592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:54.957593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:54.957797 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:54.957874 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:54.958015 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 13:37:54.958097 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:37:54.976807 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35715
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:37:55.057168 28644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:37:55.057420 28644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:37:55.057461 28644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:37:55.058614 28644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:37:55.058682 28644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:37:55.058718 28644 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 13:37:55.058746 28644 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:37:55.058779 28644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:37:55.060176 28644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:35715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:37:55.060443 28644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:37:55.060664 28644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:37:55.065744 28649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:37:55.065769 28650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:55.065826 28644 server_base.cc:1034] running on GCE node
W20260924 13:37:55.065793 28652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:37:55.066103 28644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:37:55.066299 28644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:37:55.067471 28644 hybrid_clock.cc:648] HybridClock initialized: now 1790257075067456 us; error 28 us; skew 500 ppm
Sep 24 13:37:55 dist-test-slave-m5z1 krb5kdc[28247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257075, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:37:55.069921 28644 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:37:55.070422 28644 webserver.cc:460] Webserver started at http://127.25.240.67:46865/ using document root <none> and password file <none>
I20260924 13:37:55.070660 28644 fs_manager.cc:359] Metadata directory not provided
I20260924 13:37:55.070729 28644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:37:55.070878 28644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:37:55.071931 28644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "37aebf7f708e4e41b91b0efdae71555c"
format_stamp: "Formatted at 2026-09-24 13:37:55 on dist-test-slave-m5z1"
server_key: "deb5fa1fc4f0a3ba745008e11449096a"
server_key_iv: "f4b5b399f800171e7737651072ff943d"
server_key_version: "encryptionkey@0"
I20260924 13:37:55.072306 28644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "37aebf7f708e4e41b91b0efdae71555c"
format_stamp: "Formatted at 2026-09-24 13:37:55 on dist-test-slave-m5z1"
server_key: "deb5fa1fc4f0a3ba745008e11449096a"
server_key_iv: "f4b5b399f800171e7737651072ff943d"
server_key_version: "encryptionkey@0"
I20260924 13:37:55.073671 28644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:37:55.074438 28659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:55.074608 28644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:55.074689 28644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "37aebf7f708e4e41b91b0efdae71555c"
format_stamp: "Formatted at 2026-09-24 13:37:55 on dist-test-slave-m5z1"
server_key: "deb5fa1fc4f0a3ba745008e11449096a"
server_key_iv: "f4b5b399f800171e7737651072ff943d"
server_key_version: "encryptionkey@0"
I20260924 13:37:55.074740 28644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:37:55.088645 28644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:37:55.090438 28644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:37:55.090545 28644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:37:55.090974 28644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:37:55.091013 28644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:55.091037 28644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:37:55.091074 28644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:55.097440 28644 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:45531
I20260924 13:37:55.097503 28774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:45531 every 8 connection(s)
I20260924 13:37:55.097808 28644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:37:55 dist-test-slave-m5z1 krb5kdc[28247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257075, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:55.101358 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.091429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35715 (local address 127.25.240.67:50663)
0924 13:37:55.091598 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.091637 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.092106 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.092431 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.092435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.092793 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:55.093070 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.093076 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.094153 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.094155 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.095570 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.095573 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.096092 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.096093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.096238 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.096833 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:55.096843 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:55.098451 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:55.100108 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:55.100119 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:55.100121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:55.100388 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:55.100608 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:55.100608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:55.100609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:55.100700 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:55.100917 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:55.100920 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:55.101022 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.101102 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.101213 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260924 13:37:55.101367 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.091452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:50663 (local address 127.25.240.126:35715)
0924 13:37:55.091528 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.091530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.091712 (+ 182us) server_negotiation.cc:407] Connection header received
0924 13:37:55.092241 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.092243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.092258 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.092346 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:55.093174 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.094048 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.095663 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.095997 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.096076 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.098540 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:55.098541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:55.098541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:55.098547 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:55.100031 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:55.100482 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:55.100483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:55.100484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:55.100542 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:55.100756 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:55.100757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:55.100757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:55.100865 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:55.100916 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.101132 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.101201 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:37:55.101418 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 28644
I20260924 13:37:55.101502 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:37:55.101742 26561 external_mini_cluster.cc:1428] Setting key f49fd035eeda89905e7a22cb3e632340
I20260924 13:37:55.102056 28775 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35715
I20260924 13:37:55.102154 28775 heartbeater.cc:464] Registering TS with master...
I20260924 13:37:55.102327 28775 heartbeater.cc:511] Master 127.25.240.126:35715 requested a full tablet report, sending...
I20260924 13:37:55.102730 28291 ts_manager.cc:194] Registered new tserver with Master: 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531)
I20260924 13:37:55.103209 28291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:50663
I20260924 13:37:55.108160 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.103032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:44427 (local address 127.25.240.126:35715)
0924 13:37:55.103083 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.103085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.103135 (+ 50us) server_negotiation.cc:407] Connection header received
0924 13:37:55.103206 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.103207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.103217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.103271 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:55.103613 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.104077 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.104780 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.104986 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.105075 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.105951 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:55.105953 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:55.105953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:55.105960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:55.107069 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:55.107400 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:55.107401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:55.107402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:55.107456 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:55.107668 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:55.107668 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:55.107669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:55.107778 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:55.107852 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.108000 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.108077 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 13:37:55.108173 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.102965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35715 (local address 127.25.240.67:44427)
0924 13:37:55.103081 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.103093 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.103151 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.103276 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.103277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.103472 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:55.103518 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.103519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.104166 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.104167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.104693 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.104695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.105074 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.105075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.105149 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.105548 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:55.105551 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:55.105865 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:55.107152 (+ 1287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:55.107153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:55.107154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:55.107325 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:55.107530 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:55.107531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:55.107531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:55.107595 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:55.107844 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:55.107845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:55.107887 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.107975 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.108072 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260924 13:37:55.114171 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:37:55 dist-test-slave-m5z1 krb5kdc[28247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:37:55 dist-test-slave-m5z1 krb5kdc[28247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:55.135780 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.129205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.25.240.126:35715)
0924 13:37:55.129278 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.129280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.129289 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:37:55.129312 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.129313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.129323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.129395 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:55.129997 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.130474 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.131211 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.131399 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.131480 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.133511 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:55.133512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:55.133513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:55.133520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:55.134711 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:55.135036 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:55.135037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:55.135037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:55.135095 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:55.135289 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:55.135290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:55.135290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:55.135398 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:55.135506 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.135605 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.135681 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:37:55.137166 28291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:37:55.138768 28291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:55.145130 28790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.140863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:45531 (local address 127.0.0.1:57596)
0924 13:37:55.141124 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.141135 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.141191 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.141593 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.141596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.141598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.141685 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.141687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.142975 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.142976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.144105 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.144108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.144796 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.144797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.144880 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.144892 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.144989 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.145044 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":60,"spinlock_wait_cycles":768,"thread_start_us":62,"threads_started":1}
I20260924 13:37:55.145134 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.141030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57596 (local address 127.25.240.67:45531)
0924 13:37:55.141398 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.141401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.141409 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:37:55.141438 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.141440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.141451 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.141554 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.141774 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.142892 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.144195 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.144700 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.144783 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.144840 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.145006 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.145043 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":89,"threads_started":1}
I20260924 13:37:55.145256 28791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.140875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32893 (local address 127.0.0.1:48836)
0924 13:37:55.141259 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.141267 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.141306 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.141881 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.141882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.141884 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.141927 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.141929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.142807 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.142808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.144132 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.144134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.145035 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.145036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.145091 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.145095 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.145124 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.145166 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":40,"thread_start_us":45,"threads_started":1}
I20260924 13:37:55.145645 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.141096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.25.240.65:32893)
0924 13:37:55.141708 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.141712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.141722 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:37:55.141759 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.141761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.141777 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.141880 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.142016 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.142720 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.144232 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.144947 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.145377 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.145447 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.145472 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.145511 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":83,"threads_started":1}
I20260924 13:37:55.145934 28709 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9c3d63ed5a43ed8baec12e97515605 (DEFAULT_TABLE table=test-table [id=86caa0c5d89049759f1b47474641aabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:55.146369 28709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9c3d63ed5a43ed8baec12e97515605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.146323 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9c3d63ed5a43ed8baec12e97515605 (DEFAULT_TABLE table=test-table [id=86caa0c5d89049759f1b47474641aabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:55.146620 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.140863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:41795 (local address 127.0.0.1:60904)
0924 13:37:55.141025 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.141077 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.141144 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.141593 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.141596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.141601 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.141685 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.141688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.143888 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.143889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.144930 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.144933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.146386 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.146387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.146454 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.146457 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.146509 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.146548 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":5,"thread_start_us":63,"threads_started":1}
I20260924 13:37:55.146677 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9c3d63ed5a43ed8baec12e97515605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.146862 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.141226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.25.240.66:41795)
0924 13:37:55.141393 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.141397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.141407 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:37:55.141450 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.141452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.141464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.141768 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.141832 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.142892 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.145032 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.146291 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.146372 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.146543 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.146571 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.146609 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260924 13:37:55.148888 28795 tablet_bootstrap.cc:492] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Bootstrap starting.
I20260924 13:37:55.149540 28556 tablet_service.cc:1467] Processing CreateTablet for tablet 0c9c3d63ed5a43ed8baec12e97515605 (DEFAULT_TABLE table=test-table [id=86caa0c5d89049759f1b47474641aabc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:55.149704 28796 tablet_bootstrap.cc:492] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Bootstrap starting.
I20260924 13:37:55.149935 28556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c9c3d63ed5a43ed8baec12e97515605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.150096 28795 tablet_bootstrap.cc:654] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.150287 28796 tablet_bootstrap.cc:654] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.150434 28795 log.cc:824] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:55.150637 28796 log.cc:824] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:55.151217 28795 tablet_bootstrap.cc:492] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: No bootstrap required, opened a new log
I20260924 13:37:55.151299 28795 ts_tablet_manager.cc:1397] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:37:55.153079 28799 tablet_bootstrap.cc:492] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Bootstrap starting.
I20260924 13:37:55.153085 28795 raft_consensus.cc:348] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.153254 28795 raft_consensus.cc:374] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.153301 28795 raft_consensus.cc:729] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37aebf7f708e4e41b91b0efdae71555c, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.153434 28795 consensus_queue.cc:260] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.153733 28799 tablet_bootstrap.cc:654] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.153740 28795 ts_tablet_manager.cc:1428] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 13:37:55.153762 28775 heartbeater.cc:503] Master 127.25.240.126:35715 was elected leader, sending a full tablet report...
I20260924 13:37:55.154071 28799 log.cc:824] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Log is configured to *not* fsync() on all Append() calls
I20260924 13:37:55.154316 28796 tablet_bootstrap.cc:492] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: No bootstrap required, opened a new log
I20260924 13:37:55.154393 28796 ts_tablet_manager.cc:1397] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 13:37:55.155176 28799 tablet_bootstrap.cc:492] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: No bootstrap required, opened a new log
I20260924 13:37:55.155228 28799 ts_tablet_manager.cc:1397] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:37:55.156412 28796 raft_consensus.cc:348] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.156561 28796 raft_consensus.cc:374] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.156607 28796 raft_consensus.cc:729] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e6064e9a714cdbb5907457ebe6cab4, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.156726 28796 consensus_queue.cc:260] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.156867 28799 raft_consensus.cc:348] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.157012 28799 raft_consensus.cc:374] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.157009 28796 ts_tablet_manager.cc:1428] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:37:55.157040 28495 heartbeater.cc:503] Master 127.25.240.126:35715 was elected leader, sending a full tablet report...
I20260924 13:37:55.157065 28799 raft_consensus.cc:729] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7219f307dc554d5e8b508bb50c7a8ed7, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.157176 28799 consensus_queue.cc:260] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.157420 28799 ts_tablet_manager.cc:1428] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:37:55.157442 28635 heartbeater.cc:503] Master 127.25.240.126:35715 was elected leader, sending a full tablet report...
I20260924 13:37:55.172732 28800 raft_consensus.cc:482] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:55.172863 28800 raft_consensus.cc:504] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.173216 28800 leader_election.cc:288] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795), a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893)
I20260924 13:37:55.175963 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.173427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:47859 (local address 127.25.240.66:41795)
0924 13:37:55.173475 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.173477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.173502 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:37:55.173531 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.173531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.173541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.173623 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.173784 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.174298 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.175270 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.175626 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.175680 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.175687 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.175853 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.175891 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:37:55.175988 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.173468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:48161 (local address 127.25.240.65:32893)
0924 13:37:55.173524 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.173525 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.173618 (+ 93us) server_negotiation.cc:407] Connection header received
0924 13:37:55.173682 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.173683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.173694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.173752 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.173907 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.174398 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.175305 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.175640 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.175727 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.175734 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.175888 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.175926 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:37:55.175978 28668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.173369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:41795 (local address 127.25.240.67:47859)
0924 13:37:55.173422 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.173433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.173475 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.173640 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.173641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.173643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.173716 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.173718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.174369 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.174370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.175184 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.175186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.175698 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.175699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.175780 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.175785 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.175832 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.175888 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":32}
I20260924 13:37:55.176168 28804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.173394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32893 (local address 127.25.240.67:48161)
0924 13:37:55.173551 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.173560 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.173630 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.173778 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.173779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.173780 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.173842 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.173843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.174472 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.174473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.175194 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.175195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.175742 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.175747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.175799 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.175802 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.175872 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.175909 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260924 13:37:55.176335 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9c3d63ed5a43ed8baec12e97515605" candidate_uuid: "37aebf7f708e4e41b91b0efdae71555c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" is_pre_election: true
I20260924 13:37:55.176484 28589 raft_consensus.cc:2454] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37aebf7f708e4e41b91b0efdae71555c in term 0.
I20260924 13:37:55.176473 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9c3d63ed5a43ed8baec12e97515605" candidate_uuid: "37aebf7f708e4e41b91b0efdae71555c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" is_pre_election: true
I20260924 13:37:55.176590 28449 raft_consensus.cc:2454] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37aebf7f708e4e41b91b0efdae71555c in term 0.
I20260924 13:37:55.176697 28661 leader_election.cc:302] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37aebf7f708e4e41b91b0efdae71555c, 7219f307dc554d5e8b508bb50c7a8ed7; no voters:
I20260924 13:37:55.176819 28800 raft_consensus.cc:2781] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:55.176872 28800 raft_consensus.cc:482] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:55.176894 28800 raft_consensus.cc:3037] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.177714 28800 raft_consensus.cc:504] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.177850 28800 leader_election.cc:288] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [CANDIDATE]: Term 1 election: Requested vote from peers 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795), a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893)
I20260924 13:37:55.178007 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9c3d63ed5a43ed8baec12e97515605" candidate_uuid: "37aebf7f708e4e41b91b0efdae71555c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e6064e9a714cdbb5907457ebe6cab4"
I20260924 13:37:55.178088 28449 raft_consensus.cc:3037] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.178741 28449 raft_consensus.cc:2454] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37aebf7f708e4e41b91b0efdae71555c in term 1.
I20260924 13:37:55.178865 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c9c3d63ed5a43ed8baec12e97515605" candidate_uuid: "37aebf7f708e4e41b91b0efdae71555c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7219f307dc554d5e8b508bb50c7a8ed7"
I20260924 13:37:55.178954 28663 leader_election.cc:302] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37aebf7f708e4e41b91b0efdae71555c, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:55.178946 28589 raft_consensus.cc:3037] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.179585 28589 raft_consensus.cc:2454] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37aebf7f708e4e41b91b0efdae71555c in term 1.
I20260924 13:37:55.179685 28800 raft_consensus.cc:2781] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:55.179826 28800 raft_consensus.cc:686] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 1 LEADER]: Becoming Leader. State: Replica: 37aebf7f708e4e41b91b0efdae71555c, State: Running, Role: LEADER
I20260924 13:37:55.179929 28800 consensus_queue.cc:237] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } }
I20260924 13:37:55.180720 28292 catalog_manager.cc:5495] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c reported cstate change: term changed from 0 to 1, leader changed from <none> to 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67). New cstate: current_term: 1 leader_uuid: "37aebf7f708e4e41b91b0efdae71555c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } health_report { overall_health: UNKNOWN } } }
W20260924 13:37:55.198359 28636 tablet.cc:2369] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 13:37:55 dist-test-slave-m5z1 krb5kdc[28247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257074, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:55.212607 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.206173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35715 (local address 127.0.0.1:41314)
0924 13:37:55.206318 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.206327 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.206387 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.206619 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.206620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.206914 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:37:55.206960 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.206962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.207559 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.207560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.208078 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.208079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.208376 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.208377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.208417 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.208987 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:37:55.208992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:37:55.210354 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:37:55.211666 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:55.211673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:55.211674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:55.211858 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:55.212027 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:37:55.212027 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:37:55.212027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:37:55.212080 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:37:55.212275 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:37:55.212313 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:37:55.212381 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.212447 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.212536 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260924 13:37:55.212713 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.206243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.25.240.126:35715)
0924 13:37:55.206322 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.206324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.206390 (+ 66us) server_negotiation.cc:407] Connection header received
0924 13:37:55.206437 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.206437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.206447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.206489 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:55.207031 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.207491 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.208145 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.208315 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.208362 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.210427 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:55.210427 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:55.210428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:55.210433 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:55.211594 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:55.211920 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:55.211921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:55.211921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:55.211968 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:55.212129 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:55.212130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:55.212130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:55.212223 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:55.212270 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.212467 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.212588 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":54}
I20260924 13:37:55.214769 28292 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:41314:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 13:37:55.214993 28292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:55.216513 28556 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab3cbcec0849929ead032ff0ae4c51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70a837c377654479ac313e90462fee57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:37:55.216544 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab3cbcec0849929ead032ff0ae4c51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70a837c377654479ac313e90462fee57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:37:55.216681 28556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab3cbcec0849929ead032ff0ae4c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.216725 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab3cbcec0849929ead032ff0ae4c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.217867 28799 tablet_bootstrap.cc:492] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: Bootstrap starting.
I20260924 13:37:55.218152 28709 tablet_service.cc:1467] Processing CreateTablet for tablet 37ab3cbcec0849929ead032ff0ae4c51 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70a837c377654479ac313e90462fee57]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:37:55.218266 28709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37ab3cbcec0849929ead032ff0ae4c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.218488 28799 tablet_bootstrap.cc:654] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.218786 28796 tablet_bootstrap.cc:492] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: Bootstrap starting.
I20260924 13:37:55.219161 28799 tablet_bootstrap.cc:492] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: No bootstrap required, opened a new log
I20260924 13:37:55.219203 28799 ts_tablet_manager.cc:1397] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:55.219357 28799 raft_consensus.cc:348] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.219398 28795 tablet_bootstrap.cc:492] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: Bootstrap starting.
I20260924 13:37:55.219439 28799 raft_consensus.cc:374] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.219463 28799 raft_consensus.cc:729] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7219f307dc554d5e8b508bb50c7a8ed7, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.219462 28796 tablet_bootstrap.cc:654] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.219513 28799 consensus_queue.cc:260] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.219641 28799 ts_tablet_manager.cc:1428] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:55.219720 28803 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.219846 28795 tablet_bootstrap.cc:654] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.219834 28803 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:55.220135 28796 tablet_bootstrap.cc:492] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: No bootstrap required, opened a new log
I20260924 13:37:55.220197 28796 ts_tablet_manager.cc:1397] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:55.220345 28796 raft_consensus.cc:348] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.220417 28796 raft_consensus.cc:374] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.220436 28796 raft_consensus.cc:729] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e6064e9a714cdbb5907457ebe6cab4, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.220484 28796 consensus_queue.cc:260] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.220526 28795 tablet_bootstrap.cc:492] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: No bootstrap required, opened a new log
I20260924 13:37:55.220567 28795 ts_tablet_manager.cc:1397] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:37:55.220571 28796 ts_tablet_manager.cc:1428] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:55.220635 28802 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.220705 28802 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:55.220708 28795 raft_consensus.cc:348] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.220791 28795 raft_consensus.cc:374] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.220819 28795 raft_consensus.cc:729] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37aebf7f708e4e41b91b0efdae71555c, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.220871 28795 consensus_queue.cc:260] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.220971 28795 ts_tablet_manager.cc:1428] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:55.221053 28805 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.221171 28805 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 13:37:55.295547 28496 tablet.cc:2369] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:37:55.326018 28802 raft_consensus.cc:482] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:55.326112 28802 raft_consensus.cc:504] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.326517 28802 leader_election.cc:288] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795), 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531)
I20260924 13:37:55.329543 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.326779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:52823 (local address 127.25.240.66:41795)
0924 13:37:55.326844 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.326846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.326946 (+ 100us) server_negotiation.cc:407] Connection header received
0924 13:37:55.327021 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.327023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.327032 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.327104 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.327232 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.327713 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.328767 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.329201 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.329281 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.329289 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.329442 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.329476 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:37:55.329578 28816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.326702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:41795 (local address 127.25.240.65:52823)
0924 13:37:55.326858 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.326869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.326965 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.327109 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.327110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.327112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.327171 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.327173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.327802 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.327803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.328676 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.328679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.329280 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.329280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.329347 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.329351 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.329432 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.329475 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260924 13:37:55.329644 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.326771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:53627 (local address 127.25.240.67:45531)
0924 13:37:55.326835 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.326838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.326951 (+ 113us) server_negotiation.cc:407] Connection header received
0924 13:37:55.327037 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.327037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.327053 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.327117 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.327307 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.327760 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.328770 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.329259 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.329337 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.329345 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.329500 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.329534 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:37:55.329749 28817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.326705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:45531 (local address 127.25.240.65:53627)
0924 13:37:55.326868 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.326878 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.326963 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.327142 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.327143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.327145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.327232 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.327234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.327859 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.327860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.328680 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.328683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.329337 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.329338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.329418 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.329427 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.329494 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.329534 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":15,"thread_start_us":70,"threads_started":1}
I20260924 13:37:55.329792 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab3cbcec0849929ead032ff0ae4c51" candidate_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" is_pre_election: true
I20260924 13:37:55.329885 28589 raft_consensus.cc:2454] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e6064e9a714cdbb5907457ebe6cab4 in term 0.
I20260924 13:37:55.330008 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab3cbcec0849929ead032ff0ae4c51" candidate_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37aebf7f708e4e41b91b0efdae71555c" is_pre_election: true
I20260924 13:37:55.330049 28381 leader_election.cc:302] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7219f307dc554d5e8b508bb50c7a8ed7, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:55.330113 28729 raft_consensus.cc:2454] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e6064e9a714cdbb5907457ebe6cab4 in term 0.
I20260924 13:37:55.330190 28802 raft_consensus.cc:2781] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:55.330236 28802 raft_consensus.cc:482] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:55.330258 28802 raft_consensus.cc:3037] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.330930 28802 raft_consensus.cc:504] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.331089 28802 leader_election.cc:288] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [CANDIDATE]: Term 1 election: Requested vote from peers 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795), 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531)
I20260924 13:37:55.331244 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab3cbcec0849929ead032ff0ae4c51" candidate_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37aebf7f708e4e41b91b0efdae71555c"
I20260924 13:37:55.331272 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37ab3cbcec0849929ead032ff0ae4c51" candidate_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7219f307dc554d5e8b508bb50c7a8ed7"
I20260924 13:37:55.331328 28729 raft_consensus.cc:3037] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.331348 28589 raft_consensus.cc:3037] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.331956 28589 raft_consensus.cc:2454] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e6064e9a714cdbb5907457ebe6cab4 in term 1.
I20260924 13:37:55.332033 28729 raft_consensus.cc:2454] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e6064e9a714cdbb5907457ebe6cab4 in term 1.
I20260924 13:37:55.332250 28380 leader_election.cc:302] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37aebf7f708e4e41b91b0efdae71555c, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:55.332352 28802 raft_consensus.cc:2781] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:55.332472 28802 raft_consensus.cc:686] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 LEADER]: Becoming Leader. State: Replica: a1e6064e9a714cdbb5907457ebe6cab4, State: Running, Role: LEADER
I20260924 13:37:55.332571 28802 consensus_queue.cc:237] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.332945 28818 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: TxnStatusTablet state changed. Reason: New leader a1e6064e9a714cdbb5907457ebe6cab4. Latest consensus state: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.333038 28818 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:37:55.333227 28820 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 13:37:55.333370 28820 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 13:37:55.333317 28292 catalog_manager.cc:5495] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65). New cstate: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } health_report { overall_health: UNKNOWN } } }
I20260924 13:37:55.339300 28812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.336502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32893 (local address 127.0.0.1:48842)
0924 13:37:55.336577 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.336594 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.336649 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.336929 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.336930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.336931 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:55.336974 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.336976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.337850 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.337851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.338393 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.338393 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.338838 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.338839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.338885 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.338900 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.339175 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.339182 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.339211 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.339246 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:37:55.339308 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.336720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48842 (local address 127.25.240.65:32893)
0924 13:37:55.336780 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.336782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.336791 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:37:55.336820 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.336821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.336838 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.336920 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:55.337041 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.337777 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.338472 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.338762 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.338836 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.338988 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.339109 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.339191 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.339215 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.339235 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:37:55.341190 28589 raft_consensus.cc:1261] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Refusing update from remote peer a1e6064e9a714cdbb5907457ebe6cab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:55.341301 28729 raft_consensus.cc:1261] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Refusing update from remote peer a1e6064e9a714cdbb5907457ebe6cab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:55.341488 28818 consensus_queue.cc:1035] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:55.341606 28802 consensus_queue.cc:1035] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:55.342705 28805 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: TxnStatusTablet state changed. Reason: New leader a1e6064e9a714cdbb5907457ebe6cab4. Latest consensus state: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.342789 28805 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:55.342939 28803 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: TxnStatusTablet state changed. Reason: New leader a1e6064e9a714cdbb5907457ebe6cab4. Latest consensus state: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.343053 28803 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:55.344080 28818 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.344180 28818 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:37:55.344400 28818 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.344480 28818 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P a1e6064e9a714cdbb5907457ebe6cab4: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:37:55.344987 28803 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.345065 28803 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P 7219f307dc554d5e8b508bb50c7a8ed7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:55.345144 28805 tablet_replica.cc:439] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } } }
I20260924 13:37:55.345234 28805 tablet_replica.cc:442] T 37ab3cbcec0849929ead032ff0ae4c51 P 37aebf7f708e4e41b91b0efdae71555c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:37:55.345490 28825 mvcc.cc:205] Tried to move back new op lower bound from 7332892980594913280 to 7332892980562784256. Current Snapshot: MvccSnapshot[applied={T|T < 7332892980594913280}]
I20260924 13:37:55.346264 28824 mvcc.cc:205] Tried to move back new op lower bound from 7332892980594913280 to 7332892980562784256. Current Snapshot: MvccSnapshot[applied={T|T < 7332892980594913280}]
W20260924 13:37:55.348333 28776 tablet.cc:2369] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:37:55.348742 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.345996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.25.240.67:45531)
0924 13:37:55.346038 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.346040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.346049 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:37:55.346104 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.346105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.346122 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.346193 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:55.346337 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.347051 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.347807 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.348118 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.348186 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.348359 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.348470 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.348558 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.348654 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.348675 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:37:55.353631 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.351258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:42935 (local address 127.25.240.65:32893)
0924 13:37:55.351298 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.351300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.351325 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:37:55.351386 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.351386 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.351393 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.351461 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:55.351607 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.352039 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.352882 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.353088 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.353169 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.353357 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.353388 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.353455 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.353514 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.353531 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:37:55.353631 28781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.351210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32893 (local address 127.25.240.67:42935)
0924 13:37:55.351269 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.351281 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.351339 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.351463 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.351464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.351465 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:55.351539 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.351540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.352135 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.352136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.352790 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.352791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.353171 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.353172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.353240 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.353257 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.353456 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.353458 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.353510 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.353550 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 13:37:55.355770 28589 raft_consensus.cc:1261] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Refusing update from remote peer 37aebf7f708e4e41b91b0efdae71555c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:55.355804 28449 raft_consensus.cc:1261] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Refusing update from remote peer 37aebf7f708e4e41b91b0efdae71555c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:55.356009 28800 consensus_queue.cc:1035] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:55.356180 28800 consensus_queue.cc:1035] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:55.357545 28823 mvcc.cc:205] Tried to move back new op lower bound from 7332892980655583232 to 7332892979937665024. Current Snapshot: MvccSnapshot[applied={T|T < 7332892980655583232}]
I20260924 13:37:55.364833 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.361909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:45531 (local address 127.25.240.65:36581)
0924 13:37:55.362145 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.362155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.362206 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.362318 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.362319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.362319 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:55.362590 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.362591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.363469 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.363469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.364038 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.364040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.364494 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.364495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.364536 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.364549 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.364713 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.364714 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.364743 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.364771 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":73,"threads_started":1}
I20260924 13:37:55.364830 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.361984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:36581 (local address 127.25.240.67:45531)
0924 13:37:55.362044 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.362046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.362214 (+ 168us) server_negotiation.cc:407] Connection header received
0924 13:37:55.362250 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.362251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.362261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.362317 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:55.362652 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.363387 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.364109 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.364406 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.364474 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.364625 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.364658 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.364713 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.364745 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.364763 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 13:37:55.373196 28292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:55.373315 28292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:55.374056 28292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: Sending DeleteTablet for 3 replicas of tablet 0c9c3d63ed5a43ed8baec12e97515605
I20260924 13:37:55.374382 28709 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9c3d63ed5a43ed8baec12e97515605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:55 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:57596
I20260924 13:37:55.374622 28841 tablet_replica.cc:330] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: stopping tablet replica
I20260924 13:37:55.374696 28429 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9c3d63ed5a43ed8baec12e97515605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:55 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:48836
I20260924 13:37:55.374729 28841 raft_consensus.cc:2229] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:37:55.374850 28841 raft_consensus.cc:2258] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:55.375193 28556 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c9c3d63ed5a43ed8baec12e97515605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:55 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:60904
I20260924 13:37:55.375293 28841 ts_tablet_manager.cc:1905] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:55.375649 28292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:37:55.375785 28292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:55.377038 28709 tablet_service.cc:1467] Processing CreateTablet for tablet 37bb423d146847f2883555cc96532c34 (DEFAULT_TABLE table=test-table [id=1b904ae016494d9bb4f71a02d2e22ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:55.377066 28841 ts_tablet_manager.cc:1918] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:37:55.377068 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 37bb423d146847f2883555cc96532c34 (DEFAULT_TABLE table=test-table [id=1b904ae016494d9bb4f71a02d2e22ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:55.377133 28841 log.cc:1196] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/0c9c3d63ed5a43ed8baec12e97515605
I20260924 13:37:55.377140 28709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bb423d146847f2883555cc96532c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.377179 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bb423d146847f2883555cc96532c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.377421 28841 ts_tablet_manager.cc:1939] T 0c9c3d63ed5a43ed8baec12e97515605 P 37aebf7f708e4e41b91b0efdae71555c: Deleting consensus metadata
I20260924 13:37:55.378356 28795 tablet_bootstrap.cc:492] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: Bootstrap starting.
I20260924 13:37:55.378472 28276 catalog_manager.cc:4841] TS 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531): tablet 0c9c3d63ed5a43ed8baec12e97515605 (table test-table [id=86caa0c5d89049759f1b47474641aabc]) successfully deleted
I20260924 13:37:55.378756 28795 tablet_bootstrap.cc:654] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.379088 28796 tablet_bootstrap.cc:492] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: Bootstrap starting.
I20260924 13:37:55.379390 28556 tablet_service.cc:1467] Processing CreateTablet for tablet 37bb423d146847f2883555cc96532c34 (DEFAULT_TABLE table=test-table [id=1b904ae016494d9bb4f71a02d2e22ff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:55.379475 28796 tablet_bootstrap.cc:654] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.379522 28556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37bb423d146847f2883555cc96532c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:55.380335 28796 tablet_bootstrap.cc:492] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: No bootstrap required, opened a new log
I20260924 13:37:55.380401 28796 ts_tablet_manager.cc:1397] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:55.380549 28796 raft_consensus.cc:348] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.380661 28796 raft_consensus.cc:374] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.380692 28796 raft_consensus.cc:729] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e6064e9a714cdbb5907457ebe6cab4, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.380731 28796 consensus_queue.cc:260] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.380776 28799 tablet_bootstrap.cc:492] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: Bootstrap starting.
I20260924 13:37:55.380831 28796 ts_tablet_manager.cc:1428] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:55.381150 28799 tablet_bootstrap.cc:654] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:55.381860 28799 tablet_bootstrap.cc:492] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: No bootstrap required, opened a new log
I20260924 13:37:55.381910 28799 ts_tablet_manager.cc:1397] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:55.382050 28799 raft_consensus.cc:348] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.382131 28799 raft_consensus.cc:374] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.382153 28799 raft_consensus.cc:729] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7219f307dc554d5e8b508bb50c7a8ed7, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.382210 28799 consensus_queue.cc:260] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.382301 28799 ts_tablet_manager.cc:1428] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:37:55.382685 28842 tablet_replica.cc:330] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: stopping tablet replica
I20260924 13:37:55.382833 28843 tablet_replica.cc:330] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: stopping tablet replica
I20260924 13:37:55.382937 28843 raft_consensus.cc:2229] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:55.382994 28843 raft_consensus.cc:2258] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:55.383152 28842 raft_consensus.cc:2229] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:55.383210 28842 raft_consensus.cc:2258] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:55.383590 28843 ts_tablet_manager.cc:1905] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:55.383617 28842 ts_tablet_manager.cc:1905] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:55.385383 28842 ts_tablet_manager.cc:1918] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:37:55.385387 28843 ts_tablet_manager.cc:1918] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:37:55.385454 28843 log.cc:1196] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/0c9c3d63ed5a43ed8baec12e97515605
I20260924 13:37:55.385577 28842 log.cc:1196] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/0c9c3d63ed5a43ed8baec12e97515605
I20260924 13:37:55.385739 28843 ts_tablet_manager.cc:1939] T 0c9c3d63ed5a43ed8baec12e97515605 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting consensus metadata
I20260924 13:37:55.385958 28842 ts_tablet_manager.cc:1939] T 0c9c3d63ed5a43ed8baec12e97515605 P a1e6064e9a714cdbb5907457ebe6cab4: Deleting consensus metadata
I20260924 13:37:55.386130 28277 catalog_manager.cc:4841] TS 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795): tablet 0c9c3d63ed5a43ed8baec12e97515605 (table test-table [id=86caa0c5d89049759f1b47474641aabc]) successfully deleted
I20260924 13:37:55.386492 28279 catalog_manager.cc:4841] TS a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893): tablet 0c9c3d63ed5a43ed8baec12e97515605 (table test-table [id=86caa0c5d89049759f1b47474641aabc]) successfully deleted
I20260924 13:37:55.390085 28795 tablet_bootstrap.cc:492] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: No bootstrap required, opened a new log
I20260924 13:37:55.390158 28795 ts_tablet_manager.cc:1397] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.001s
I20260924 13:37:55.390323 28795 raft_consensus.cc:348] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.390404 28795 raft_consensus.cc:374] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:55.390432 28795 raft_consensus.cc:729] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37aebf7f708e4e41b91b0efdae71555c, State: Initialized, Role: FOLLOWER
I20260924 13:37:55.390473 28795 consensus_queue.cc:260] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.390578 28795 ts_tablet_manager.cc:1428] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 13:37:55.465237 28803 raft_consensus.cc:482] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:55.465344 28803 raft_consensus.cc:504] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.465711 28803 leader_election.cc:288] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893), 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531)
I20260924 13:37:55.469743 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.466048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:47489 (local address 127.25.240.65:32893)
0924 13:37:55.466095 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.466098 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.466228 (+ 130us) server_negotiation.cc:407] Connection header received
0924 13:37:55.466336 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.466337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.466348 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.466378 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.466599 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.467325 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.468818 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.469343 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.469386 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.469395 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.469638 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.469672 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:37:55.469758 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.466072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:58709 (local address 127.25.240.67:45531)
0924 13:37:55.466116 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.466118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.466296 (+ 178us) server_negotiation.cc:407] Connection header received
0924 13:37:55.466401 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.466403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.466412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.466441 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:37:55.466646 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.467361 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.468821 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.469342 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.469394 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.469402 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.469643 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.469678 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:37:55.470005 28847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.465883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32893 (local address 127.25.240.66:47489)
0924 13:37:55.466161 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.466172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.466256 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.466444 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.466445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.466447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.466516 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.466518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.467425 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.467426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.468719 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.468721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.469450 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.469451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.469525 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.469529 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.469887 (+ 358us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.469930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":75,"thread_start_us":59,"threads_started":1}
I20260924 13:37:55.470211 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bb423d146847f2883555cc96532c34" candidate_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" is_pre_election: true
I20260924 13:37:55.470302 28449 raft_consensus.cc:2454] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7219f307dc554d5e8b508bb50c7a8ed7 in term 0.
I20260924 13:37:55.470439 28848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.465916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:45531 (local address 127.25.240.66:58709)
0924 13:37:55.466225 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.466237 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.466324 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.466503 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.466504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.466507 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:37:55.466573 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.466575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.467466 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.467466 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.468717 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.468720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.469461 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.469462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.469532 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.469536 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.470225 (+ 689us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.470267 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":53,"threads_started":1}
I20260924 13:37:55.470450 28523 leader_election.cc:302] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7219f307dc554d5e8b508bb50c7a8ed7, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:55.470800 28803 raft_consensus.cc:2781] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:55.470832 28803 raft_consensus.cc:482] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:55.470851 28803 raft_consensus.cc:3037] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.470850 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bb423d146847f2883555cc96532c34" candidate_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37aebf7f708e4e41b91b0efdae71555c" is_pre_election: true
I20260924 13:37:55.470929 28729 raft_consensus.cc:2454] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7219f307dc554d5e8b508bb50c7a8ed7 in term 0.
I20260924 13:37:55.471685 28803 raft_consensus.cc:504] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.471814 28803 leader_election.cc:288] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [CANDIDATE]: Term 1 election: Requested vote from peers a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893), 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531)
I20260924 13:37:55.472043 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bb423d146847f2883555cc96532c34" candidate_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37aebf7f708e4e41b91b0efdae71555c"
I20260924 13:37:55.472072 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37bb423d146847f2883555cc96532c34" candidate_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e6064e9a714cdbb5907457ebe6cab4"
I20260924 13:37:55.472131 28729 raft_consensus.cc:3037] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.472141 28449 raft_consensus.cc:3037] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:55.472890 28449 raft_consensus.cc:2454] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7219f307dc554d5e8b508bb50c7a8ed7 in term 1.
I20260924 13:37:55.473026 28729 raft_consensus.cc:2454] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7219f307dc554d5e8b508bb50c7a8ed7 in term 1.
I20260924 13:37:55.473050 28523 leader_election.cc:302] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7219f307dc554d5e8b508bb50c7a8ed7, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:55.473204 28803 raft_consensus.cc:2781] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:55.473310 28803 raft_consensus.cc:686] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 LEADER]: Becoming Leader. State: Replica: 7219f307dc554d5e8b508bb50c7a8ed7, State: Running, Role: LEADER
I20260924 13:37:55.473372 28803 consensus_queue.cc:237] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:55.474020 28292 catalog_manager.cc:5495] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66). New cstate: current_term: 1 leader_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } health_report { overall_health: UNKNOWN } } }
I20260924 13:37:55.482473 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.479498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.25.240.66:41795)
0924 13:37:55.479539 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.479542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.479554 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:37:55.479591 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.479591 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.479616 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.479704 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:55.479846 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.480552 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.481574 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.481872 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.481938 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.482109 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.482223 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.482289 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.482362 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.482382 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:37:55.488040 28793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.485120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:39221 (local address 127.25.240.65:32893)
0924 13:37:55.485157 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.485159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.485201 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:37:55.485266 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.485266 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.485276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.485328 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:55.485458 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.486181 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.487267 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.487522 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.487591 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.487794 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.487825 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.487875 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.487948 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.487968 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:37:55.488257 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.484990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32893 (local address 127.25.240.66:39221)
0924 13:37:55.485140 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.485152 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.485215 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.485332 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.485332 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.485334 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:55.485392 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.485393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.486281 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.486282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.487165 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.487168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.487611 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.487613 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.487684 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.487700 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.487893 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.487896 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.487946 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.487992 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260924 13:37:55.490345 28729 raft_consensus.cc:1261] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Refusing update from remote peer 7219f307dc554d5e8b508bb50c7a8ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:55.490346 28449 raft_consensus.cc:1261] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Refusing update from remote peer 7219f307dc554d5e8b508bb50c7a8ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:55.490634 28849 consensus_queue.cc:1035] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:55.490780 28803 consensus_queue.cc:1035] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:55.497793 28838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.495353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:41795 (local address 127.25.240.65:48143)
0924 13:37:55.495391 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:55.495403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:55.495445 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:55.495680 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:55.495681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:55.495683 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:55.495718 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.495719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.496373 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.496374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.496921 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:55.496923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.497392 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.497393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:55.497449 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.497454 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.497651 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.497653 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:37:55.497700 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:37:55.497747 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 13:37:55.497804 28794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:55.495517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:48143 (local address 127.25.240.66:41795)
0924 13:37:55.495578 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:55.495580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:55.495588 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:37:55.495608 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:55.495609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:55.495619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:55.495665 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:55.495763 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.496280 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.497008 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:55.497301 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:55.497359 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:55.497516 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:55.497552 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:55.497650 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:55.497702 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 13:37:55.497724 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 13:37:55.498096 28825 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 13:37:57.016211 28291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:57.016310 28291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:57.017068 28291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: Sending DeleteTablet for 3 replicas of tablet 37bb423d146847f2883555cc96532c34
I20260924 13:37:57.017360 28429 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bb423d146847f2883555cc96532c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:57 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:48836
I20260924 13:37:57.017360 28709 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bb423d146847f2883555cc96532c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:57 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:57596
I20260924 13:37:57.017581 28556 tablet_service.cc:1514] Processing DeleteTablet for tablet 37bb423d146847f2883555cc96532c34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:57 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:60904
I20260924 13:37:57.017783 28881 tablet_replica.cc:330] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: stopping tablet replica
I20260924 13:37:57.017848 28881 raft_consensus.cc:2229] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:37:57.017942 28880 tablet_replica.cc:330] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: stopping tablet replica
I20260924 13:37:57.017999 28880 raft_consensus.cc:2229] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:57.018062 28880 raft_consensus.cc:2258] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:57.018184 28881 raft_consensus.cc:2258] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:57.018488 28880 ts_tablet_manager.cc:1905] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:57.019626 28879 tablet_replica.cc:330] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: stopping tablet replica
I20260924 13:37:57.019773 28879 raft_consensus.cc:2229] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:57.019876 28879 raft_consensus.cc:2258] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:57.020416 28880 ts_tablet_manager.cc:1918] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:37:57.020475 28880 log.cc:1196] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/37bb423d146847f2883555cc96532c34
I20260924 13:37:57.020536 28881 ts_tablet_manager.cc:1905] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:57.020743 28880 ts_tablet_manager.cc:1939] T 37bb423d146847f2883555cc96532c34 P 37aebf7f708e4e41b91b0efdae71555c: Deleting consensus metadata
I20260924 13:37:57.021062 28276 catalog_manager.cc:4841] TS 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531): tablet 37bb423d146847f2883555cc96532c34 (table test-table [id=1b904ae016494d9bb4f71a02d2e22ff3]) successfully deleted
I20260924 13:37:57.021661 28879 ts_tablet_manager.cc:1905] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:57.022450 28881 ts_tablet_manager.cc:1918] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:37:57.022517 28881 log.cc:1196] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/37bb423d146847f2883555cc96532c34
I20260924 13:37:57.022755 28881 ts_tablet_manager.cc:1939] T 37bb423d146847f2883555cc96532c34 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting consensus metadata
I20260924 13:37:57.023036 28277 catalog_manager.cc:4841] TS 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795): tablet 37bb423d146847f2883555cc96532c34 (table test-table [id=1b904ae016494d9bb4f71a02d2e22ff3]) successfully deleted
I20260924 13:37:57.023270 28879 ts_tablet_manager.cc:1918] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:37:57.023319 28879 log.cc:1196] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/37bb423d146847f2883555cc96532c34
I20260924 13:37:57.023469 28879 ts_tablet_manager.cc:1939] T 37bb423d146847f2883555cc96532c34 P a1e6064e9a714cdbb5907457ebe6cab4: Deleting consensus metadata
I20260924 13:37:57.023845 28279 catalog_manager.cc:4841] TS a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893): tablet 37bb423d146847f2883555cc96532c34 (table test-table [id=1b904ae016494d9bb4f71a02d2e22ff3]) successfully deleted
Sep 24 13:37:57 dist-test-slave-m5z1 krb5kdc[28247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:37:57 dist-test-slave-m5z1 krb5kdc[28247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:37:57.045646 28889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:57.035102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.25.240.126:35715)
0924 13:37:57.035257 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:57.035259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:57.035303 (+ 44us) server_negotiation.cc:407] Connection header received
0924 13:37:57.035405 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:57.035406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:57.035417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:57.035484 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:37:57.036047 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.036574 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.037322 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.037542 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.037652 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:57.043169 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:37:57.043170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:37:57.043171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:37:57.043179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:37:57.044449 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:57.044805 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:57.044806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:57.044807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:57.044867 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:37:57.045075 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:37:57.045076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:37:57.045076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:37:57.045198 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:37:57.045308 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:57.045415 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 13:37:57.045499 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260924 13:37:57.047091 28291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:37:57.047273 28291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:57.048606 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 1cec1b69814e4540adf91c7ba85f300a (DEFAULT_TABLE table=test-table [id=add727113cb94aa68941dde34df2cf76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:57.048656 28556 tablet_service.cc:1467] Processing CreateTablet for tablet 1cec1b69814e4540adf91c7ba85f300a (DEFAULT_TABLE table=test-table [id=add727113cb94aa68941dde34df2cf76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:57.048707 28709 tablet_service.cc:1467] Processing CreateTablet for tablet 1cec1b69814e4540adf91c7ba85f300a (DEFAULT_TABLE table=test-table [id=add727113cb94aa68941dde34df2cf76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:57.048761 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cec1b69814e4540adf91c7ba85f300a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:57.048790 28556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cec1b69814e4540adf91c7ba85f300a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:57.048804 28709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cec1b69814e4540adf91c7ba85f300a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:57.049818 28890 tablet_bootstrap.cc:492] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: Bootstrap starting.
I20260924 13:37:57.050297 28890 tablet_bootstrap.cc:654] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:57.050333 28892 tablet_bootstrap.cc:492] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: Bootstrap starting.
I20260924 13:37:57.050721 28892 tablet_bootstrap.cc:654] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:57.051033 28891 tablet_bootstrap.cc:492] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: Bootstrap starting.
I20260924 13:37:57.051045 28890 tablet_bootstrap.cc:492] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: No bootstrap required, opened a new log
I20260924 13:37:57.051115 28890 ts_tablet_manager.cc:1397] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:57.051272 28890 raft_consensus.cc:348] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.051353 28890 raft_consensus.cc:374] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:57.051395 28890 raft_consensus.cc:729] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e6064e9a714cdbb5907457ebe6cab4, State: Initialized, Role: FOLLOWER
I20260924 13:37:57.051374 28891 tablet_bootstrap.cc:654] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:57.051458 28890 consensus_queue.cc:260] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.051566 28890 ts_tablet_manager.cc:1428] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:57.051767 28892 tablet_bootstrap.cc:492] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: No bootstrap required, opened a new log
I20260924 13:37:57.051823 28892 ts_tablet_manager.cc:1397] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:37:57.051959 28892 raft_consensus.cc:348] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.051978 28891 tablet_bootstrap.cc:492] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: No bootstrap required, opened a new log
I20260924 13:37:57.052031 28891 ts_tablet_manager.cc:1397] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:57.052037 28892 raft_consensus.cc:374] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:57.052063 28892 raft_consensus.cc:729] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7219f307dc554d5e8b508bb50c7a8ed7, State: Initialized, Role: FOLLOWER
I20260924 13:37:57.052140 28892 consensus_queue.cc:260] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.052148 28891 raft_consensus.cc:348] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.052222 28891 raft_consensus.cc:374] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:57.052275 28891 raft_consensus.cc:729] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37aebf7f708e4e41b91b0efdae71555c, State: Initialized, Role: FOLLOWER
I20260924 13:37:57.052336 28891 consensus_queue.cc:260] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.052440 28892 ts_tablet_manager.cc:1428] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:37:57.052594 28891 ts_tablet_manager.cc:1428] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:57.244125 28834 raft_consensus.cc:482] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:57.244227 28834 raft_consensus.cc:504] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.244391 28834 leader_election.cc:288] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795), 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531)
I20260924 13:37:57.244619 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cec1b69814e4540adf91c7ba85f300a" candidate_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" is_pre_election: true
I20260924 13:37:57.244664 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cec1b69814e4540adf91c7ba85f300a" candidate_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37aebf7f708e4e41b91b0efdae71555c" is_pre_election: true
I20260924 13:37:57.244735 28589 raft_consensus.cc:2454] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e6064e9a714cdbb5907457ebe6cab4 in term 0.
I20260924 13:37:57.244740 28729 raft_consensus.cc:2454] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1e6064e9a714cdbb5907457ebe6cab4 in term 0.
I20260924 13:37:57.244901 28381 leader_election.cc:302] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7219f307dc554d5e8b508bb50c7a8ed7, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:57.245033 28834 raft_consensus.cc:2781] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:57.245079 28834 raft_consensus.cc:482] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:57.245095 28834 raft_consensus.cc:3037] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:57.245947 28834 raft_consensus.cc:504] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.246089 28834 leader_election.cc:288] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [CANDIDATE]: Term 1 election: Requested vote from peers 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795), 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531)
I20260924 13:37:57.246241 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cec1b69814e4540adf91c7ba85f300a" candidate_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7219f307dc554d5e8b508bb50c7a8ed7"
I20260924 13:37:57.246311 28589 raft_consensus.cc:3037] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:57.246260 28729 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cec1b69814e4540adf91c7ba85f300a" candidate_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37aebf7f708e4e41b91b0efdae71555c"
I20260924 13:37:57.246343 28729 raft_consensus.cc:3037] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:57.246924 28589 raft_consensus.cc:2454] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e6064e9a714cdbb5907457ebe6cab4 in term 1.
I20260924 13:37:57.246939 28729 raft_consensus.cc:2454] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1e6064e9a714cdbb5907457ebe6cab4 in term 1.
I20260924 13:37:57.247098 28381 leader_election.cc:302] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7219f307dc554d5e8b508bb50c7a8ed7, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:57.247218 28834 raft_consensus.cc:2781] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:57.247272 28834 raft_consensus.cc:686] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 LEADER]: Becoming Leader. State: Replica: a1e6064e9a714cdbb5907457ebe6cab4, State: Running, Role: LEADER
I20260924 13:37:57.247318 28834 consensus_queue.cc:237] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.247946 28292 catalog_manager.cc:5495] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65). New cstate: current_term: 1 leader_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } health_report { overall_health: UNKNOWN } } }
I20260924 13:37:57.279917 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:57.277595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42922 (local address 127.25.240.65:32893)
0924 13:37:57.277752 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:57.277754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:57.277761 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:37:57.277780 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:57.277780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:57.277790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:57.277847 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:57.277984 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.278466 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.279296 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.279483 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.279537 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:57.279663 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:57.279691 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:57.279729 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:57.279815 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 13:37:57.279831 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260924 13:37:57.284207 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:57.282259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:42293 (local address 127.25.240.65:32893)
0924 13:37:57.282320 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:57.282322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:57.282371 (+ 49us) server_negotiation.cc:407] Connection header received
0924 13:37:57.282421 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:57.282421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:57.282430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:57.282476 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:57.282565 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.282999 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.283674 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.283844 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.283894 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:57.284009 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:57.284031 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:57.284066 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:57.284120 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 13:37:57.284134 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:37:57.284269 28902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:57.282149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32893 (local address 127.25.240.65:42293)
0924 13:37:57.282326 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:57.282334 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:37:57.282382 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:37:57.282485 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:37:57.282486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:37:57.282487 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:37:57.282521 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:57.282522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.283059 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.283060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:57.283606 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:37:57.283607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.283910 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.283910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:37:57.283949 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:57.283952 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:37:57.284083 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:37:57.284084 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:37:57.284119 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 13:37:57.284161 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
I20260924 13:37:57.286218 28589 raft_consensus.cc:1261] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Refusing update from remote peer a1e6064e9a714cdbb5907457ebe6cab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:57.286273 28729 raft_consensus.cc:1261] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Refusing update from remote peer a1e6064e9a714cdbb5907457ebe6cab4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:57.286429 28903 consensus_queue.cc:1035] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:57.286517 28860 consensus_queue.cc:1035] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:57.296756 28292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:57.296861 28292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:57.297466 28292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: Sending DeleteTablet for 3 replicas of tablet 1cec1b69814e4540adf91c7ba85f300a
I20260924 13:37:57.297726 28556 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cec1b69814e4540adf91c7ba85f300a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:57 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:60904
I20260924 13:37:57.297734 28429 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cec1b69814e4540adf91c7ba85f300a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:57 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:48836
I20260924 13:37:57.297824 28879 tablet_replica.cc:330] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: stopping tablet replica
I20260924 13:37:57.297818 28709 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cec1b69814e4540adf91c7ba85f300a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:57 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:57596
I20260924 13:37:57.297844 28881 tablet_replica.cc:330] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: stopping tablet replica
I20260924 13:37:57.297873 28879 raft_consensus.cc:2229] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:37:57.297896 28881 raft_consensus.cc:2229] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:57.297899 28880 tablet_replica.cc:330] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: stopping tablet replica
I20260924 13:37:57.297928 28881 raft_consensus.cc:2258] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:57.297955 28879 raft_consensus.cc:2258] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:57.297950 28880 raft_consensus.cc:2229] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:57.298015 28880 raft_consensus.cc:2258] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:57.298295 28879 ts_tablet_manager.cc:1905] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:57.298352 28881 ts_tablet_manager.cc:1905] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:57.300060 28292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 13:37:57.300225 28881 ts_tablet_manager.cc:1918] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:37:57.300225 28879 ts_tablet_manager.cc:1918] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W20260924 13:37:57.300241 28292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:37:57.300273 28881 log.cc:1196] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/1cec1b69814e4540adf91c7ba85f300a
I20260924 13:37:57.300279 28879 log.cc:1196] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/1cec1b69814e4540adf91c7ba85f300a
I20260924 13:37:57.300503 28881 ts_tablet_manager.cc:1939] T 1cec1b69814e4540adf91c7ba85f300a P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting consensus metadata
I20260924 13:37:57.300532 28879 ts_tablet_manager.cc:1939] T 1cec1b69814e4540adf91c7ba85f300a P a1e6064e9a714cdbb5907457ebe6cab4: Deleting consensus metadata
I20260924 13:37:57.300887 28279 catalog_manager.cc:4841] TS a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893): tablet 1cec1b69814e4540adf91c7ba85f300a (table test-table [id=add727113cb94aa68941dde34df2cf76]) successfully deleted
I20260924 13:37:57.300895 28277 catalog_manager.cc:4841] TS 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795): tablet 1cec1b69814e4540adf91c7ba85f300a (table test-table [id=add727113cb94aa68941dde34df2cf76]) successfully deleted
I20260924 13:37:57.301618 28556 tablet_service.cc:1467] Processing CreateTablet for tablet 3617fb50d4ce4aa39f79c0855eb675f1 (DEFAULT_TABLE table=test-table [id=3a3b4fcce21142d2b92b28baa2469797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:57.301620 28429 tablet_service.cc:1467] Processing CreateTablet for tablet 3617fb50d4ce4aa39f79c0855eb675f1 (DEFAULT_TABLE table=test-table [id=3a3b4fcce21142d2b92b28baa2469797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:57.301673 28709 tablet_service.cc:1467] Processing CreateTablet for tablet 3617fb50d4ce4aa39f79c0855eb675f1 (DEFAULT_TABLE table=test-table [id=3a3b4fcce21142d2b92b28baa2469797]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:37:57.301728 28556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3617fb50d4ce4aa39f79c0855eb675f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:57.301738 28429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3617fb50d4ce4aa39f79c0855eb675f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:57.301765 28709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3617fb50d4ce4aa39f79c0855eb675f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:37:57.302569 28890 tablet_bootstrap.cc:492] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4: Bootstrap starting.
I20260924 13:37:57.302927 28890 tablet_bootstrap.cc:654] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:57.302920 28892 tablet_bootstrap.cc:492] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: Bootstrap starting.
I20260924 13:37:57.303195 28891 tablet_bootstrap.cc:492] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: Bootstrap starting.
I20260924 13:37:57.303220 28892 tablet_bootstrap.cc:654] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:57.303200 28880 ts_tablet_manager.cc:1905] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:57.303547 28891 tablet_bootstrap.cc:654] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: Neither blocks nor log segments found. Creating new log.
I20260924 13:37:57.304064 28891 tablet_bootstrap.cc:492] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: No bootstrap required, opened a new log
I20260924 13:37:57.304111 28891 ts_tablet_manager.cc:1397] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:37:57.304219 28891 raft_consensus.cc:348] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.304294 28891 raft_consensus.cc:374] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:57.304317 28891 raft_consensus.cc:729] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37aebf7f708e4e41b91b0efdae71555c, State: Initialized, Role: FOLLOWER
I20260924 13:37:57.304366 28891 consensus_queue.cc:260] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.304430 28890 tablet_bootstrap.cc:492] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4: No bootstrap required, opened a new log
I20260924 13:37:57.304459 28891 ts_tablet_manager.cc:1428] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:57.304468 28890 ts_tablet_manager.cc:1397] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:37:57.304569 28890 raft_consensus.cc:348] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.304636 28890 raft_consensus.cc:374] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:57.304661 28890 raft_consensus.cc:729] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1e6064e9a714cdbb5907457ebe6cab4, State: Initialized, Role: FOLLOWER
I20260924 13:37:57.304708 28890 consensus_queue.cc:260] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.304795 28880 ts_tablet_manager.cc:1918] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:37:57.304811 28890 ts_tablet_manager.cc:1428] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:57.304847 28880 log.cc:1196] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/1cec1b69814e4540adf91c7ba85f300a
I20260924 13:37:57.305069 28880 ts_tablet_manager.cc:1939] T 1cec1b69814e4540adf91c7ba85f300a P 37aebf7f708e4e41b91b0efdae71555c: Deleting consensus metadata
I20260924 13:37:57.305330 28276 catalog_manager.cc:4841] TS 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531): tablet 1cec1b69814e4540adf91c7ba85f300a (table test-table [id=add727113cb94aa68941dde34df2cf76]) successfully deleted
I20260924 13:37:57.305629 28892 tablet_bootstrap.cc:492] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: No bootstrap required, opened a new log
I20260924 13:37:57.305704 28892 ts_tablet_manager.cc:1397] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260924 13:37:57.305851 28892 raft_consensus.cc:348] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.305920 28892 raft_consensus.cc:374] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:37:57.305948 28892 raft_consensus.cc:729] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7219f307dc554d5e8b508bb50c7a8ed7, State: Initialized, Role: FOLLOWER
I20260924 13:37:57.305992 28892 consensus_queue.cc:260] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.306085 28892 ts_tablet_manager.cc:1428] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:37:57.324551 28896 raft_consensus.cc:482] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:37:57.324638 28896 raft_consensus.cc:504] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.324805 28896 leader_election.cc:288] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893), 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795)
I20260924 13:37:57.324967 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3617fb50d4ce4aa39f79c0855eb675f1" candidate_uuid: "37aebf7f708e4e41b91b0efdae71555c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" is_pre_election: true
I20260924 13:37:57.324970 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3617fb50d4ce4aa39f79c0855eb675f1" candidate_uuid: "37aebf7f708e4e41b91b0efdae71555c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" is_pre_election: true
I20260924 13:37:57.325049 28589 raft_consensus.cc:2454] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37aebf7f708e4e41b91b0efdae71555c in term 0.
I20260924 13:37:57.325060 28449 raft_consensus.cc:2454] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37aebf7f708e4e41b91b0efdae71555c in term 0.
I20260924 13:37:57.325193 28663 leader_election.cc:302] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37aebf7f708e4e41b91b0efdae71555c, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:57.325307 28896 raft_consensus.cc:2781] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:37:57.325358 28896 raft_consensus.cc:482] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:37:57.325388 28896 raft_consensus.cc:3037] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:57.326195 28896 raft_consensus.cc:504] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.326347 28896 leader_election.cc:288] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [CANDIDATE]: Term 1 election: Requested vote from peers a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893), 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795)
I20260924 13:37:57.326459 28449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3617fb50d4ce4aa39f79c0855eb675f1" candidate_uuid: "37aebf7f708e4e41b91b0efdae71555c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1e6064e9a714cdbb5907457ebe6cab4"
I20260924 13:37:57.326540 28449 raft_consensus.cc:3037] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:57.326507 28589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3617fb50d4ce4aa39f79c0855eb675f1" candidate_uuid: "37aebf7f708e4e41b91b0efdae71555c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7219f307dc554d5e8b508bb50c7a8ed7"
I20260924 13:37:57.326599 28589 raft_consensus.cc:3037] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:37:57.327284 28589 raft_consensus.cc:2454] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37aebf7f708e4e41b91b0efdae71555c in term 1.
I20260924 13:37:57.327310 28449 raft_consensus.cc:2454] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37aebf7f708e4e41b91b0efdae71555c in term 1.
I20260924 13:37:57.327455 28663 leader_election.cc:302] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37aebf7f708e4e41b91b0efdae71555c, a1e6064e9a714cdbb5907457ebe6cab4; no voters:
I20260924 13:37:57.327591 28896 raft_consensus.cc:2781] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:37:57.327710 28896 raft_consensus.cc:686] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 1 LEADER]: Becoming Leader. State: Replica: 37aebf7f708e4e41b91b0efdae71555c, State: Running, Role: LEADER
I20260924 13:37:57.327761 28896 consensus_queue.cc:237] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } }
I20260924 13:37:57.328279 28292 catalog_manager.cc:5495] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c reported cstate change: term changed from 0 to 1, leader changed from <none> to 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67). New cstate: current_term: 1 leader_uuid: "37aebf7f708e4e41b91b0efdae71555c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37aebf7f708e4e41b91b0efdae71555c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 45531 } health_report { overall_health: HEALTHY } } }
I20260924 13:37:57.335832 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:57.333436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41390 (local address 127.25.240.67:45531)
0924 13:37:57.333616 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 13:37:57.333617 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:37:57.333624 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:37:57.333652 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:37:57.333653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:37:57.333662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:37:57.333735 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:37:57.333841 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.334349 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.335101 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:37:57.335328 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:37:57.335404 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:37:57.335566 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:37:57.335601 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:37:57.335659 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:37:57.335717 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 13:37:57.335738 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260924 13:37:57.337993 28449 raft_consensus.cc:1261] T 3617fb50d4ce4aa39f79c0855eb675f1 P a1e6064e9a714cdbb5907457ebe6cab4 [term 1 FOLLOWER]: Refusing update from remote peer 37aebf7f708e4e41b91b0efdae71555c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:57.337989 28589 raft_consensus.cc:1261] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Refusing update from remote peer 37aebf7f708e4e41b91b0efdae71555c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:37:57.338232 28896 consensus_queue.cc:1035] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7219f307dc554d5e8b508bb50c7a8ed7" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:37:57.338322 28910 consensus_queue.cc:1035] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1e6064e9a714cdbb5907457ebe6cab4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:37:57.343081 28861 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 13:37:58.860738 28292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:58.860837 28292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:37:58.861662 28292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c3982224b8494e4287051c2f5b1c97e5: Sending DeleteTablet for 3 replicas of tablet 3617fb50d4ce4aa39f79c0855eb675f1
I20260924 13:37:58.862010 28556 tablet_service.cc:1514] Processing DeleteTablet for tablet 3617fb50d4ce4aa39f79c0855eb675f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:58 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:60904
I20260924 13:37:58.862012 28428 tablet_service.cc:1514] Processing DeleteTablet for tablet 3617fb50d4ce4aa39f79c0855eb675f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:58 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:48836
I20260924 13:37:58.862126 28709 tablet_service.cc:1514] Processing DeleteTablet for tablet 3617fb50d4ce4aa39f79c0855eb675f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:37:58 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:57596
I20260924 13:37:58.862283 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 28364
I20260924 13:37:58.862315 28941 tablet_replica.cc:330] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: stopping tablet replica
I20260924 13:37:58.862939 28941 raft_consensus.cc:2229] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:37:58.863018 28941 raft_consensus.cc:2258] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:58.863272 28941 ts_tablet_manager.cc:1905] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:58.865216 28941 ts_tablet_manager.cc:1918] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:37:58.865268 28941 log.cc:1196] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/3617fb50d4ce4aa39f79c0855eb675f1
I20260924 13:37:58.865489 28941 ts_tablet_manager.cc:1939] T 3617fb50d4ce4aa39f79c0855eb675f1 P 37aebf7f708e4e41b91b0efdae71555c: Deleting consensus metadata
I20260924 13:37:58.865780 28940 tablet_replica.cc:330] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: stopping tablet replica
I20260924 13:37:58.865831 28276 catalog_manager.cc:4841] TS 37aebf7f708e4e41b91b0efdae71555c (127.25.240.67:45531): tablet 3617fb50d4ce4aa39f79c0855eb675f1 (table test-table [id=3a3b4fcce21142d2b92b28baa2469797]) successfully deleted
I20260924 13:37:58.865859 28940 raft_consensus.cc:2229] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:37:58.865900 28940 raft_consensus.cc:2258] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:37:58.866142 28940 ts_tablet_manager.cc:1905] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:37:58.867440 28940 ts_tablet_manager.cc:1918] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 13:37:58.867487 28940 log.cc:1196] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/3617fb50d4ce4aa39f79c0855eb675f1
I20260924 13:37:58.867753 28940 ts_tablet_manager.cc:1939] T 3617fb50d4ce4aa39f79c0855eb675f1 P 7219f307dc554d5e8b508bb50c7a8ed7: Deleting consensus metadata
I20260924 13:37:58.868054 28277 catalog_manager.cc:4841] TS 7219f307dc554d5e8b508bb50c7a8ed7 (127.25.240.66:41795): tablet 3617fb50d4ce4aa39f79c0855eb675f1 (table test-table [id=3a3b4fcce21142d2b92b28baa2469797]) successfully deleted
W20260924 13:37:58.870564 28279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:32893 (error 108)
I20260924 13:37:58.870821 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 28504
I20260924 13:37:58.871490 28943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:37:58.870921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32893 (local address 127.0.0.1:42928)
0924 13:37:58.871397 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 13:37:58.871447 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:32893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":56,"threads_started":1}
W20260924 13:37:58.871642 28279 catalog_manager.cc:4584] TS a1e6064e9a714cdbb5907457ebe6cab4 (127.25.240.65:32893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3617fb50d4ce4aa39f79c0855eb675f1: Network error: Client connection negotiation failed: client connection to 127.25.240.65:32893: connect: Connection refused (error 111)
I20260924 13:37:58.877336 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 28644
I20260924 13:37:58.882766 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 28260
2026-09-24T13:37:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5577 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:37:58 dist-test-slave-m5z1 krb5kdc[28946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:37:58 dist-test-slave-m5z1 krb5kdc[28946](info): set up 2 sockets
Sep 24 13:37:58 dist-test-slave-m5z1 krb5kdc[28946](info): commencing operation
krb5kdc: starting...
W20260924 13:37:59.925500 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:00.040266 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37309
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:37309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:00.119975 28959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:00.120210 28959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:00.120235 28959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:00.121336 28959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:00.121376 28959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:00.121389 28959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:00.121402 28959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:00.121413 28959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:00.122820 28959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:37309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:00.123011 28959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:00.123203 28959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:00.127948 28964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:00.127996 28967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:00.127948 28965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:00.128253 28959 server_base.cc:1034] running on GCE node
I20260924 13:38:00.128532 28959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:00.128819 28959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:00.129961 28959 hybrid_clock.cc:648] HybridClock initialized: now 1790257080129954 us; error 34 us; skew 500 ppm
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:00.132385 28959 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:00.132820 28959 webserver.cc:460] Webserver started at http://127.25.240.126:42695/ using document root <none> and password file <none>
I20260924 13:38:00.133046 28959 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:00.133112 28959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:00.133270 28959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:00.134436 28959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "df4997eadd9f47f783450237de5669c0"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "f5482588f8f7856f94752c4fda3f2d07"
server_key_iv: "1b4e684eb5f4ebc9b26d8776b5a6f64d"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.134819 28959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "df4997eadd9f47f783450237de5669c0"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "f5482588f8f7856f94752c4fda3f2d07"
server_key_iv: "1b4e684eb5f4ebc9b26d8776b5a6f64d"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.136103 28959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:38:00.136803 28974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.136971 28959 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:00.137077 28959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "df4997eadd9f47f783450237de5669c0"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "f5482588f8f7856f94752c4fda3f2d07"
server_key_iv: "1b4e684eb5f4ebc9b26d8776b5a6f64d"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.137152 28959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:00.153800 28959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:00.155718 28959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:00.155881 28959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:00.159742 28959 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:37309
I20260924 13:38:00.159777 29026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:37309 every 8 connection(s)
I20260924 13:38:00.160116 28959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:00.160983 29027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:00.163506 29027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0: Bootstrap starting.
I20260924 13:38:00.164188 29027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:00.164363 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 28959
I20260924 13:38:00.164424 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:00.164418 29027 log.cc:824] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:00.164548 26561 external_mini_cluster.cc:1428] Setting key df620fa2d2ddaf45be5f0665f015072d
I20260924 13:38:00.165238 29027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0: No bootstrap required, opened a new log
I20260924 13:38:00.167963 29027 raft_consensus.cc:348] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4997eadd9f47f783450237de5669c0" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37309 } }
I20260924 13:38:00.168135 29027 raft_consensus.cc:374] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:00.168169 29027 raft_consensus.cc:729] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df4997eadd9f47f783450237de5669c0, State: Initialized, Role: FOLLOWER
I20260924 13:38:00.168283 29027 consensus_queue.cc:260] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4997eadd9f47f783450237de5669c0" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37309 } }
I20260924 13:38:00.168351 29027 raft_consensus.cc:388] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:00.168378 29027 raft_consensus.cc:482] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:00.168421 29027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:00.169016 29027 raft_consensus.cc:504] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4997eadd9f47f783450237de5669c0" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37309 } }
I20260924 13:38:00.169142 29027 leader_election.cc:302] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df4997eadd9f47f783450237de5669c0; no voters:
I20260924 13:38:00.169327 29027 leader_election.cc:288] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:00.169365 29032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:00.169530 29032 raft_consensus.cc:686] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [term 1 LEADER]: Becoming Leader. State: Replica: df4997eadd9f47f783450237de5669c0, State: Running, Role: LEADER
I20260924 13:38:00.169626 29027 sys_catalog.cc:564] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:00.169749 29032 consensus_queue.cc:237] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4997eadd9f47f783450237de5669c0" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37309 } }
I20260924 13:38:00.170207 29034 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df4997eadd9f47f783450237de5669c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4997eadd9f47f783450237de5669c0" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37309 } } }
I20260924 13:38:00.170382 29034 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:00.171068 29033 sys_catalog.cc:455] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df4997eadd9f47f783450237de5669c0. Latest consensus state: current_term: 1 leader_uuid: "df4997eadd9f47f783450237de5669c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df4997eadd9f47f783450237de5669c0" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37309 } } }
I20260924 13:38:00.171252 29033 sys_catalog.cc:458] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0 [sys.catalog]: This master's current role is: LEADER
W20260924 13:38:00.171480 29047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:38:00.171537 29047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:00.171649 29041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:00.171867 29041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:00.173939 29041 catalog_manager.cc:1347] Generated new cluster ID: 8ee2df6794624de1aee9081ddda9c0c0
I20260924 13:38:00.173995 29041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:00.175762 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:00.165033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.25.240.126:37309)
0924 13:38:00.165196 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:00.165205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:00.165244 (+ 39us) server_negotiation.cc:407] Connection header received
0924 13:38:00.165766 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:00.165773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:00.165819 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:00.165927 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:00.166702 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.167662 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.168695 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.169088 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.169220 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:00.172100 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:00.172104 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:00.172105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:00.172117 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:00.174124 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:00.174515 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:00.174518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:00.174519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:00.174625 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:00.174838 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:00.174839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:00.174839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:00.175149 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:00.175311 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:00.175381 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 13:38:00.175519 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":94,"threads_started":1}
I20260924 13:38:00.189266 29041 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:00.190124 29041 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:00.198378 29041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df4997eadd9f47f783450237de5669c0: Generated new TSK 0
I20260924 13:38:00.198580 29041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:38:00.228276 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37309
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:00.311342 29053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:00.311573 29053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:00.311611 29053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:00.312793 29053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:00.312858 29053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:00.312901 29053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:00.314371 29053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:00.314628 29053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:00.314854 29053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:00.319744 29058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:00.319790 29061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:00.319770 29059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:00.320011 29053 server_base.cc:1034] running on GCE node
I20260924 13:38:00.320267 29053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:00.320510 29053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:00.321617 29053 hybrid_clock.cc:648] HybridClock initialized: now 1790257080321599 us; error 26 us; skew 500 ppm
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:00.324153 29053 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:38:00.324633 29053 webserver.cc:460] Webserver started at http://127.25.240.65:33563/ using document root <none> and password file <none>
I20260924 13:38:00.324856 29053 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:00.324922 29053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:00.325070 29053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:00.326143 29053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "af417c9922fb46538f746094d8f47d3c"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "3343031ed3356ca0266ab4dc4164701d"
server_key_iv: "d031965c9c19c5c2990770e25d4c5b33"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.326505 29053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "af417c9922fb46538f746094d8f47d3c"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "3343031ed3356ca0266ab4dc4164701d"
server_key_iv: "d031965c9c19c5c2990770e25d4c5b33"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.327804 29053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:00.328518 29068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.328693 29053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:00.328790 29053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "af417c9922fb46538f746094d8f47d3c"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "3343031ed3356ca0266ab4dc4164701d"
server_key_iv: "d031965c9c19c5c2990770e25d4c5b33"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.328873 29053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:00.336462 29053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:00.338824 29053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:00.338980 29053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:00.339212 29053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:00.339515 29053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:00.339569 29053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.339610 29053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:00.339639 29053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.345286 29053 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:38639
I20260924 13:38:00.345355 29181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:38639 every 8 connection(s)
I20260924 13:38:00.345734 29053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:00.353080 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 29053
I20260924 13:38:00.353197 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:00.353389 26561 external_mini_cluster.cc:1428] Setting key 19692934f91f468a0c409ef66b4e5a37
I20260924 13:38:00.354859 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:00.346478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:44381 (local address 127.25.240.126:37309)
0924 13:38:00.346560 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:00.346563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:00.346731 (+ 168us) server_negotiation.cc:407] Connection header received
0924 13:38:00.347253 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:00.347255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:00.347267 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:00.347325 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:00.348252 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.348775 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.349851 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.350062 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.350148 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:00.352111 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:00.352112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:00.352113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:00.352121 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:00.353395 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:00.353897 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:00.353898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:00.353899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:00.353963 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:00.354223 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:00.354224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:00.354224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:00.354344 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:00.354420 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:00.354682 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 13:38:00.354768 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:38:00.355038 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:00.346437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37309 (local address 127.25.240.65:44381)
0924 13:38:00.346616 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:00.346653 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:00.347152 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:00.347410 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:00.347413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:00.347960 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:00.348150 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:00.348155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.348873 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.348875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:00.349749 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:00.349752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.350162 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.350163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:00.350284 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:00.350714 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:00.350722 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:00.352013 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:00.353483 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:00.353490 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:00.353492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:00.353811 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:00.354029 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:00.354030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:00.354030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:00.354148 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:00.354411 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:00.354417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:00.354554 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 13:38:00.354661 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 13:38:00.354810 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260924 13:38:00.355670 29182 heartbeater.cc:347] Connected to a master server at 127.25.240.126:37309
I20260924 13:38:00.355779 29182 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:00.355952 29182 heartbeater.cc:511] Master 127.25.240.126:37309 requested a full tablet report, sending...
I20260924 13:38:00.356527 28991 ts_manager.cc:194] Registered new tserver with Master: af417c9922fb46538f746094d8f47d3c (127.25.240.65:38639)
I20260924 13:38:00.357438 28991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:44381
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:38:00.378705 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37309
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:00.461272 29187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:00.461473 29187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:00.461496 29187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:00.462710 29187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:00.462754 29187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:00.462781 29187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:00.464210 29187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:00.464453 29187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:00.464704 29187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:00.469862 29193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:00.469911 29187 server_base.cc:1034] running on GCE node
W20260924 13:38:00.469884 29192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:00.470043 29195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:00.470240 29187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:00.470520 29187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:00.471657 29187 hybrid_clock.cc:648] HybridClock initialized: now 1790257080471641 us; error 27 us; skew 500 ppm
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:00.474153 29187 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:38:00.474575 29187 webserver.cc:460] Webserver started at http://127.25.240.66:41101/ using document root <none> and password file <none>
I20260924 13:38:00.474802 29187 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:00.474867 29187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:00.475014 29187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:00.476043 29187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "91801f9c818c4c59916091651b002a9f"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "29d5074f653a7188a92ae2e3b5a9060b"
server_key_iv: "a66bb36d6ca2cfc75f24a2f7bbc403d6"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.476399 29187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "91801f9c818c4c59916091651b002a9f"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "29d5074f653a7188a92ae2e3b5a9060b"
server_key_iv: "a66bb36d6ca2cfc75f24a2f7bbc403d6"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.477757 29187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:38:00.478438 29202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.478600 29187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.478703 29187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "91801f9c818c4c59916091651b002a9f"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "29d5074f653a7188a92ae2e3b5a9060b"
server_key_iv: "a66bb36d6ca2cfc75f24a2f7bbc403d6"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.478773 29187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:00.504658 29187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:00.507138 29187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:00.507313 29187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:00.507551 29187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:00.507855 29187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:00.507913 29187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.507952 29187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:00.507978 29187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.513742 29187 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:38049
I20260924 13:38:00.513794 29315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:38049 every 8 connection(s)
I20260924 13:38:00.514115 29187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:00.522732 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:00.514833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:32881 (local address 127.25.240.126:37309)
0924 13:38:00.514900 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:00.514902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:00.515071 (+ 169us) server_negotiation.cc:407] Connection header received
0924 13:38:00.515539 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:00.515540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:00.515549 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:00.515608 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:00.516485 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.516992 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.518117 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.518294 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.518345 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:00.520414 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:00.520415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:00.520415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:00.520422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:00.521625 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:00.521975 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:00.521976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:00.521977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:00.522026 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:00.522224 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:00.522224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:00.522224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:00.522319 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:00.522366 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:00.522598 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 13:38:00.522668 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:38:00.522895 29318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:00.514781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37309 (local address 127.25.240.66:32881)
0924 13:38:00.514939 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:00.514987 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:00.515459 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:00.515662 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:00.515665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:00.516220 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:00.516406 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:00.516410 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.517120 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.517121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:00.518047 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:00.518050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.518363 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.518364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:00.518459 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:00.518984 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:00.519000 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:00.520329 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:00.521707 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:00.521712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:00.521714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:00.521913 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:00.522082 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:00.522083 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:00.522083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:00.522172 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:00.522369 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:00.522373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:00.522475 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 13:38:00.522581 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 13:38:00.522697 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260924 13:38:00.523425 29316 heartbeater.cc:347] Connected to a master server at 127.25.240.126:37309
I20260924 13:38:00.523516 29316 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:00.523509 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 29187
I20260924 13:38:00.523600 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:00.523718 29316 heartbeater.cc:511] Master 127.25.240.126:37309 requested a full tablet report, sending...
I20260924 13:38:00.523806 26561 external_mini_cluster.cc:1428] Setting key 03ff2d654f105ba28300c8c99f832c21
I20260924 13:38:00.524153 28991 ts_manager.cc:194] Registered new tserver with Master: 91801f9c818c4c59916091651b002a9f (127.25.240.66:38049)
I20260924 13:38:00.524688 28991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:32881
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:38:00.548429 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37309
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:00.627805 29321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:00.628000 29321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:00.628021 29321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:00.629113 29321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:00.629150 29321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:00.629175 29321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:00.630535 29321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:00.630718 29321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:00.630877 29321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:00.635802 29327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:00.635736 29329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:00.635740 29326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:00.636153 29321 server_base.cc:1034] running on GCE node
I20260924 13:38:00.636322 29321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:00.636500 29321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:00.637645 29321 hybrid_clock.cc:648] HybridClock initialized: now 1790257080637621 us; error 26 us; skew 500 ppm
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:00.640038 29321 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:38:00.640484 29321 webserver.cc:460] Webserver started at http://127.25.240.67:40167/ using document root <none> and password file <none>
I20260924 13:38:00.640759 29321 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:00.640827 29321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:00.640964 29321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:00.642037 29321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "ee3e5ca4ff214945ad003176ab7700f3"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "81652b0fb2bbc65ecca9be8905eb6877"
server_key_iv: "4a13885a97bd0bbd7ccb7ff4964c1a82"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.642431 29321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "ee3e5ca4ff214945ad003176ab7700f3"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "81652b0fb2bbc65ecca9be8905eb6877"
server_key_iv: "4a13885a97bd0bbd7ccb7ff4964c1a82"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.643711 29321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:00.644394 29336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.644567 29321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:00.644671 29321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "ee3e5ca4ff214945ad003176ab7700f3"
format_stamp: "Formatted at 2026-09-24 13:38:00 on dist-test-slave-m5z1"
server_key: "81652b0fb2bbc65ecca9be8905eb6877"
server_key_iv: "4a13885a97bd0bbd7ccb7ff4964c1a82"
server_key_version: "encryptionkey@0"
I20260924 13:38:00.644753 29321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:00.674175 29321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:00.677058 29321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:00.677206 29321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:00.677445 29321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:00.677845 29321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:00.677893 29321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.677925 29321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:00.677942 29321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:00.684715 29321 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:44837
I20260924 13:38:00.684790 29449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:44837 every 8 connection(s)
I20260924 13:38:00.685158 29321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[28946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257080, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:00.693203 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 29321
I20260924 13:38:00.693313 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:00.693506 26561 external_mini_cluster.cc:1428] Setting key ab4f01259891ec74e68394a32fc1425d
I20260924 13:38:00.695957 29031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:00.686628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:36825 (local address 127.25.240.126:37309)
0924 13:38:00.686680 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:00.686682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:00.686710 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:38:00.687091 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:00.687093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:00.687105 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:00.687172 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:00.688401 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.689080 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.690655 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.690883 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.690950 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:00.693174 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:00.693175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:00.693176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:00.693184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:00.694830 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:00.695225 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:00.695226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:00.695227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:00.695277 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:00.695459 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:00.695460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:00.695460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:00.695560 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:00.695609 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:00.695837 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 13:38:00.695895 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:38:00.696136 29452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:00.686118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37309 (local address 127.25.240.67:36825)
0924 13:38:00.686329 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:00.686380 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:00.686980 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:00.687246 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:00.687250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:00.688000 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:00.688304 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:00.688310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.689198 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.689199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:00.690515 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:00.690518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:00.690963 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:00.690964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:00.691069 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:00.691615 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:00.691623 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:00.693021 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:00.694909 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:00.694915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:00.694917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:00.695156 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:00.695318 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:00.695319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:00.695319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:00.695410 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:00.695616 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:00.695621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:00.695731 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 13:38:00.695814 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:38:00.695928 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260924 13:38:00.696609 29450 heartbeater.cc:347] Connected to a master server at 127.25.240.126:37309
I20260924 13:38:00.696684 29450 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:00.696866 29450 heartbeater.cc:511] Master 127.25.240.126:37309 requested a full tablet report, sending...
I20260924 13:38:00.697230 28991 ts_manager.cc:194] Registered new tserver with Master: ee3e5ca4ff214945ad003176ab7700f3 (127.25.240.67:44837)
I20260924 13:38:00.697618 28991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:36825
I20260924 13:38:00.705154 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 13:38:00.715849 29031 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:00.714968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.25.240.126:37309)
0924 13:38:00.715049 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:00.715051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:00.715112 (+ 61us) server_negotiation.cc:407] Connection header received
0924 13:38:00.715160 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:00.715160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:00.715170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:00.715226 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:00.715782 (+ 556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58912: BlockingRecv error: recv got EOF from 127.0.0.1:58912 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:38:00.716715 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 29053
I20260924 13:38:00.721072 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 29187
I20260924 13:38:00.725673 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 29321
I20260924 13:38:00.729712 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 28959
2026-09-24T13:38:00Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1845 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[29462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[29462](info): set up 2 sockets
Sep 24 13:38:00 dist-test-slave-m5z1 krb5kdc[29462](info): commencing operation
krb5kdc: starting...
W20260924 13:38:01.757690 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:01 dist-test-slave-m5z1 krb5kdc[29462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:01.870507 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35957
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:39433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:35957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:01.950095 29475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:01.950347 29475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:01.950387 29475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:01.951571 29475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:01.951640 29475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:01.951674 29475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:01.951707 29475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:01.951732 29475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:01.953263 29475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:39433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:35957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:01.953549 29475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:01.953809 29475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:01.958701 29481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:01.958865 29475 server_base.cc:1034] running on GCE node
W20260924 13:38:01.958762 29483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:01.958719 29480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:01.959199 29475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:01.959501 29475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:01.960672 29475 hybrid_clock.cc:648] HybridClock initialized: now 1790257081960659 us; error 53 us; skew 500 ppm
Sep 24 13:38:01 dist-test-slave-m5z1 krb5kdc[29462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257081, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:01.963145 29475 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:01.963647 29475 webserver.cc:460] Webserver started at http://127.25.240.126:36481/ using document root <none> and password file <none>
I20260924 13:38:01.963898 29475 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:01.963972 29475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:01.964128 29475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:01.965209 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "09faa1aedbf14b09b3a14f980f0a5736"
format_stamp: "Formatted at 2026-09-24 13:38:01 on dist-test-slave-m5z1"
server_key: "d2f0d89361a1abab4ce873c6f715177f"
server_key_iv: "3d50ea8aff5cb469d8124304e9689925"
server_key_version: "encryptionkey@0"
I20260924 13:38:01.965631 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "09faa1aedbf14b09b3a14f980f0a5736"
format_stamp: "Formatted at 2026-09-24 13:38:01 on dist-test-slave-m5z1"
server_key: "d2f0d89361a1abab4ce873c6f715177f"
server_key_iv: "3d50ea8aff5cb469d8124304e9689925"
server_key_version: "encryptionkey@0"
I20260924 13:38:01.966979 29475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:01.967718 29490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:01.967897 29475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:01.968009 29475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "09faa1aedbf14b09b3a14f980f0a5736"
format_stamp: "Formatted at 2026-09-24 13:38:01 on dist-test-slave-m5z1"
server_key: "d2f0d89361a1abab4ce873c6f715177f"
server_key_iv: "3d50ea8aff5cb469d8124304e9689925"
server_key_version: "encryptionkey@0"
I20260924 13:38:01.968080 29475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:01.982380 29475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:01.984267 29475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:01.984386 29475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:01.988164 29475 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:35957
I20260924 13:38:01.988560 29475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:01.988546 29542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:35957 every 8 connection(s)
I20260924 13:38:01.989301 29543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:01.991647 29543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736: Bootstrap starting.
I20260924 13:38:01.992239 29543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:01.992566 29543 log.cc:824] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:01.993238 29543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736: No bootstrap required, opened a new log
I20260924 13:38:01.994977 29543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09faa1aedbf14b09b3a14f980f0a5736" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35957 } }
I20260924 13:38:01.995134 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 29475
I20260924 13:38:01.995174 29543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:01.995216 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:01.995214 29543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09faa1aedbf14b09b3a14f980f0a5736, State: Initialized, Role: FOLLOWER
I20260924 13:38:01.995394 26561 external_mini_cluster.cc:1428] Setting key f8daf2b94b8b818166c259ecdd3f3d55
I20260924 13:38:01.995370 29543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09faa1aedbf14b09b3a14f980f0a5736" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35957 } }
I20260924 13:38:01.995447 29543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:01.995491 29543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:01.995545 29543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:01.996781 29543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09faa1aedbf14b09b3a14f980f0a5736" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35957 } }
I20260924 13:38:01.996876 29543 leader_election.cc:302] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09faa1aedbf14b09b3a14f980f0a5736; no voters:
I20260924 13:38:01.997077 29543 leader_election.cc:288] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:01.997385 29543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:01.997105 29548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:01.998932 29548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [term 1 LEADER]: Becoming Leader. State: Replica: 09faa1aedbf14b09b3a14f980f0a5736, State: Running, Role: LEADER
I20260924 13:38:01.999066 29548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09faa1aedbf14b09b3a14f980f0a5736" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35957 } }
I20260924 13:38:01.999558 29560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09faa1aedbf14b09b3a14f980f0a5736. Latest consensus state: current_term: 1 leader_uuid: "09faa1aedbf14b09b3a14f980f0a5736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09faa1aedbf14b09b3a14f980f0a5736" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35957 } } }
I20260924 13:38:01.999787 29549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09faa1aedbf14b09b3a14f980f0a5736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09faa1aedbf14b09b3a14f980f0a5736" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35957 } } }
I20260924 13:38:02.001051 29560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:02.001056 29549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736 [sys.catalog]: This master's current role is: LEADER
W20260924 13:38:02.000370 29562 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:38:02.001255 29562 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:38:02.001302 29565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:02.001462 29565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 13:38:01 dist-test-slave-m5z1 krb5kdc[29462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:02.003095 29565 catalog_manager.cc:1347] Generated new cluster ID: 14b30c089fc248df8d53d2f0f3b459dc
I20260924 13:38:02.003149 29565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:02.007026 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:01.995835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.25.240.126:35957)
0924 13:38:01.996080 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:01.996091 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:01.996133 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:38:01.996296 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:01.996305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:01.996373 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:01.996527 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:01.997202 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:01.998716 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:01.999701 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:02.000346 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:02.000469 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:02.003310 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:02.003314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:02.003315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:02.003327 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:02.005471 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:02.005868 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:02.005871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:02.005871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:02.005986 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:02.006208 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:02.006209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:02.006210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:02.006450 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:02.006633 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:02.006700 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 13:38:02.006850 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":116,"threads_started":1}
I20260924 13:38:02.007894 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:38:02.022110 29547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:02.018561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.25.240.126:35957)
0924 13:38:02.018620 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:02.018623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:02.018783 (+ 160us) server_negotiation.cc:407] Connection header received
0924 13:38:02.018860 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:02.018860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:02.018872 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:02.018928 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:02.019309 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:02.019809 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:02.020553 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:02.020779 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:02.020868 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:02.021620 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:02.021622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:02.021622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:02.021835 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:02.021851 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:02.021857 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:02.021924 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:02.021990 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 13:38:02.022014 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:38:02.031589 29565 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:02.032132 29565 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:02.035897 29565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09faa1aedbf14b09b3a14f980f0a5736: Generated new TSK 0
I20260924 13:38:02.036101 29565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:38:02.041368 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 29475
2026-09-24T13:38:02Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1309 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:02 dist-test-slave-m5z1 krb5kdc[29576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:02 dist-test-slave-m5z1 krb5kdc[29576](info): set up 2 sockets
Sep 24 13:38:02 dist-test-slave-m5z1 krb5kdc[29576](info): commencing operation
krb5kdc: starting...
W20260924 13:38:03.089869 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:03.208184 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:40623
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:40623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:03.288072 29589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:03.288331 29589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:03.288368 29589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:03.289525 29589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:03.289628 29589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:03.289664 29589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:03.289695 29589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:03.289716 29589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:03.291199 29589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:40623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:40623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:03.291486 29589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:03.291728 29589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:03.296701 29594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:03.296833 29589 server_base.cc:1034] running on GCE node
W20260924 13:38:03.296711 29595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:03.296716 29597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:03.297248 29589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:03.297564 29589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:03.298753 29589 hybrid_clock.cc:648] HybridClock initialized: now 1790257083298732 us; error 36 us; skew 500 ppm
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:03.301174 29589 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:03.301664 29589 webserver.cc:460] Webserver started at http://127.25.240.126:41593/ using document root <none> and password file <none>
I20260924 13:38:03.301908 29589 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:03.301976 29589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:03.302129 29589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:03.303179 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "5af8d59fa5824c3380445d80283e1e2f"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "b4fb0aa3325334d33516b91366a265a0"
server_key_iv: "affe678736bc642e11ab47b14bbcb83c"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.303534 29589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "5af8d59fa5824c3380445d80283e1e2f"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "b4fb0aa3325334d33516b91366a265a0"
server_key_iv: "affe678736bc642e11ab47b14bbcb83c"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.304804 29589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:03.305480 29604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.305663 29589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:03.305773 29589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "5af8d59fa5824c3380445d80283e1e2f"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "b4fb0aa3325334d33516b91366a265a0"
server_key_iv: "affe678736bc642e11ab47b14bbcb83c"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.305846 29589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:03.322570 29589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:03.324389 29589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:03.324553 29589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:03.328315 29589 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:40623
I20260924 13:38:03.328366 29656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:40623 every 8 connection(s)
I20260924 13:38:03.328728 29589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:03.329777 29657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:03.331993 29657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f: Bootstrap starting.
I20260924 13:38:03.332522 29657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:03.332840 29657 log.cc:824] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:03.332863 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 29589
I20260924 13:38:03.332958 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:03.333142 26561 external_mini_cluster.cc:1428] Setting key 9ed1208918791ef91f3c93394c884f8a
I20260924 13:38:03.333690 29657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f: No bootstrap required, opened a new log
I20260924 13:38:03.335347 29657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8d59fa5824c3380445d80283e1e2f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40623 } }
I20260924 13:38:03.335510 29657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:03.335556 29657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af8d59fa5824c3380445d80283e1e2f, State: Initialized, Role: FOLLOWER
I20260924 13:38:03.335687 29657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8d59fa5824c3380445d80283e1e2f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40623 } }
I20260924 13:38:03.335770 29657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:03.335813 29657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:03.335886 29657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:03.336750 29657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8d59fa5824c3380445d80283e1e2f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40623 } }
I20260924 13:38:03.336871 29657 leader_election.cc:302] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5af8d59fa5824c3380445d80283e1e2f; no voters:
I20260924 13:38:03.337059 29657 leader_election.cc:288] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:03.337106 29662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:03.337244 29662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [term 1 LEADER]: Becoming Leader. State: Replica: 5af8d59fa5824c3380445d80283e1e2f, State: Running, Role: LEADER
I20260924 13:38:03.337327 29657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:03.337327 29662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8d59fa5824c3380445d80283e1e2f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40623 } }
I20260924 13:38:03.337739 29662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5af8d59fa5824c3380445d80283e1e2f. Latest consensus state: current_term: 1 leader_uuid: "5af8d59fa5824c3380445d80283e1e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8d59fa5824c3380445d80283e1e2f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40623 } } }
I20260924 13:38:03.337821 29662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:03.337987 29663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5af8d59fa5824c3380445d80283e1e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5af8d59fa5824c3380445d80283e1e2f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40623 } } }
I20260924 13:38:03.338109 29663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:03.338762 29666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:03.339296 29666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:03.341053 29666 catalog_manager.cc:1347] Generated new cluster ID: bc6a69940b61450bbfc2e8bb35e6cd9b
I20260924 13:38:03.341182 29666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:03.345340 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:03.333677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.25.240.126:40623)
0924 13:38:03.333884 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:03.333901 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:03.333925 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:38:03.335332 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:03.335339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:03.335387 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:03.335525 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:03.336291 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.337439 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.338569 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.339228 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.339370 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:03.342072 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:03.342077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:03.342077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:03.342089 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:03.343791 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.344178 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.344187 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.344187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.344291 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.344506 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.344507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.344507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.344739 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:03.344911 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:03.345024 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 13:38:03.345168 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260924 13:38:03.353161 29666 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:03.353865 29666 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:03.367091 29666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5af8d59fa5824c3380445d80283e1e2f: Generated new TSK 0
I20260924 13:38:03.367307 29666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:38:03.400780 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40623
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:03.481115 29683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:03.481351 29683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:03.481391 29683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:03.482589 29683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:03.482658 29683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:03.482712 29683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:03.484153 29683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:03.484414 29683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:03.484652 29683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:03.489737 29691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:03.489835 29683 server_base.cc:1034] running on GCE node
W20260924 13:38:03.489692 29688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:03.489702 29689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:03.490139 29683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:03.490393 29683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:03.491552 29683 hybrid_clock.cc:648] HybridClock initialized: now 1790257083491553 us; error 38 us; skew 500 ppm
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:03.494019 29683 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:38:03.494445 29683 webserver.cc:460] Webserver started at http://127.25.240.65:34853/ using document root <none> and password file <none>
I20260924 13:38:03.494669 29683 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:03.494743 29683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:03.494895 29683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:03.496025 29683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "c72f40932ff04abc8fcedd51bf58b456"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "9cbcb874837ee4974b765af44605cb4e"
server_key_iv: "b09019220a00f8f11025b3a1d547ad85"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.496402 29683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "c72f40932ff04abc8fcedd51bf58b456"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "9cbcb874837ee4974b765af44605cb4e"
server_key_iv: "b09019220a00f8f11025b3a1d547ad85"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.497758 29683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:38:03.498435 29698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.498615 29683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:03.498714 29683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "c72f40932ff04abc8fcedd51bf58b456"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "9cbcb874837ee4974b765af44605cb4e"
server_key_iv: "b09019220a00f8f11025b3a1d547ad85"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.498785 29683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:03.510253 29683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:03.512481 29683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:03.512653 29683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:03.512871 29683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:03.513167 29683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:03.513228 29683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.513269 29683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:03.513298 29683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.518882 29683 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:41543
I20260924 13:38:03.518954 29811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:41543 every 8 connection(s)
I20260924 13:38:03.519280 29683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:03.525836 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 29683
I20260924 13:38:03.525943 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:03.526134 26561 external_mini_cluster.cc:1428] Setting key b696925ea954cebd615c70de6c2fe164
I20260924 13:38:03.528452 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:03.520024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:51101 (local address 127.25.240.126:40623)
0924 13:38:03.520089 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:03.520091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:03.520318 (+ 227us) server_negotiation.cc:407] Connection header received
0924 13:38:03.520782 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:03.520783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:03.520795 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:03.520852 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:03.521742 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.522234 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.523151 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.523357 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.523408 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:03.525358 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:03.525359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:03.525359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:03.525369 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:03.527087 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.527460 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.527462 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.527463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.527526 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.527766 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.527767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.527767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.527918 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:03.527986 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:03.528272 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 13:38:03.528362 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:38:03.528631 29814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:03.519996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40623 (local address 127.25.240.65:51101)
0924 13:38:03.520177 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:03.520218 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:03.520699 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:03.520905 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:03.520908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:03.521457 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:03.521664 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:03.521669 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.522312 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.522313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:03.523079 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:03.523080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.523427 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.523428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:03.523523 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:03.523935 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:03.523943 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:03.525268 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:03.527167 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:03.527172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:03.527174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:03.527384 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:03.527598 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:03.527598 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:03.527599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:03.527702 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:03.527980 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:03.527984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:03.528134 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 13:38:03.528244 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 13:38:03.528400 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
I20260924 13:38:03.529310 29812 heartbeater.cc:347] Connected to a master server at 127.25.240.126:40623
I20260924 13:38:03.529389 29812 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:03.529633 29812 heartbeater.cc:511] Master 127.25.240.126:40623 requested a full tablet report, sending...
I20260924 13:38:03.530160 29621 ts_manager.cc:194] Registered new tserver with Master: c72f40932ff04abc8fcedd51bf58b456 (127.25.240.65:41543)
I20260924 13:38:03.530768 29621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:51101
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:38:03.552598 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40623
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:03.632750 29817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:03.632951 29817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:03.632974 29817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:03.634112 29817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:03.634153 29817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:03.634179 29817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:03.635548 29817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:03.635730 29817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:03.635907 29817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:03.641196 29822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:03.641270 29823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:03.641270 29825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:03.641719 29817 server_base.cc:1034] running on GCE node
I20260924 13:38:03.641922 29817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:03.642108 29817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:03.643242 29817 hybrid_clock.cc:648] HybridClock initialized: now 1790257083643230 us; error 27 us; skew 500 ppm
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:03.645684 29817 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:38:03.646158 29817 webserver.cc:460] Webserver started at http://127.25.240.66:41193/ using document root <none> and password file <none>
I20260924 13:38:03.646396 29817 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:03.646441 29817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:03.646582 29817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:03.647663 29817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "d089207318994469bb6b8b96ecd8e4b8"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "1db2f4f8781823699dfc9d3b0f0405cd"
server_key_iv: "83037450f493174bd8125c882bb7f3ec"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.648033 29817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "d089207318994469bb6b8b96ecd8e4b8"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "1db2f4f8781823699dfc9d3b0f0405cd"
server_key_iv: "83037450f493174bd8125c882bb7f3ec"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.649334 29817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:03.650111 29832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.650311 29817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.650398 29817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "d089207318994469bb6b8b96ecd8e4b8"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "1db2f4f8781823699dfc9d3b0f0405cd"
server_key_iv: "83037450f493174bd8125c882bb7f3ec"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.650470 29817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:03.670523 29817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:03.672607 29817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:03.672763 29817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:03.672981 29817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:03.673286 29817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:03.673342 29817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.673384 29817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:03.673411 29817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.679198 29817 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:45809
I20260924 13:38:03.679292 29945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:45809 every 8 connection(s)
I20260924 13:38:03.679608 29817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:03.687397 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 29817
I20260924 13:38:03.687515 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:03.687708 26561 external_mini_cluster.cc:1428] Setting key 3798ded252320943b7d6b711252e2fe7
I20260924 13:38:03.688722 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:03.680339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:51227 (local address 127.25.240.126:40623)
0924 13:38:03.680390 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:03.680392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:03.680612 (+ 220us) server_negotiation.cc:407] Connection header received
0924 13:38:03.681138 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:03.681139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:03.681152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:03.681215 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:03.682154 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.682665 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.683674 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.683888 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.683968 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:03.685968 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:03.685970 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:03.685970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:03.685978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:03.687126 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.687622 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.687623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.687623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.687689 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.687990 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.687992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.687992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.688149 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:03.688219 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:03.688532 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 13:38:03.688621 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:38:03.688892 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:03.680292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40623 (local address 127.25.240.66:51227)
0924 13:38:03.680442 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:03.680500 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:03.681016 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:03.681292 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:03.681297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:03.681864 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:03.682049 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:03.682053 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.682782 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.682784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:03.683573 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:03.683575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.683971 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.683972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:03.684104 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:03.684542 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:03.684550 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:03.685853 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:03.687223 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:03.687239 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:03.687241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:03.687552 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:03.687768 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:03.687769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:03.687770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:03.687906 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:03.688218 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:03.688223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:03.688389 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 13:38:03.688509 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 13:38:03.688663 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260924 13:38:03.689424 29946 heartbeater.cc:347] Connected to a master server at 127.25.240.126:40623
I20260924 13:38:03.689503 29946 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:03.689709 29946 heartbeater.cc:511] Master 127.25.240.126:40623 requested a full tablet report, sending...
I20260924 13:38:03.690068 29621 ts_manager.cc:194] Registered new tserver with Master: d089207318994469bb6b8b96ecd8e4b8 (127.25.240.66:45809)
I20260924 13:38:03.690428 29621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:51227
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:38:03.714046 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40623
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:37099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:03.794308 29951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:03.794492 29951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:03.794514 29951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:03.795646 29951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:03.795694 29951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:03.795747 29951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:03.797128 29951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:03.797360 29951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:03.797602 29951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:03.802435 29957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:03.802425 29956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:03.802425 29959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:03.802717 29951 server_base.cc:1034] running on GCE node
I20260924 13:38:03.802906 29951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:03.803083 29951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:03.804219 29951 hybrid_clock.cc:648] HybridClock initialized: now 1790257083804203 us; error 33 us; skew 500 ppm
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:03.806743 29951 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:38:03.807102 29951 webserver.cc:460] Webserver started at http://127.25.240.67:46169/ using document root <none> and password file <none>
I20260924 13:38:03.807298 29951 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:03.807338 29951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:03.807425 29951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:03.808439 29951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "57be3fdb5e3848faafe57f62d941cbbf"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "893a99f10a024a9915538a65c3f79237"
server_key_iv: "1cb9daf12ba60bb6588ff40767c759e0"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.808821 29951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "57be3fdb5e3848faafe57f62d941cbbf"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "893a99f10a024a9915538a65c3f79237"
server_key_iv: "1cb9daf12ba60bb6588ff40767c759e0"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.810207 29951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:03.810887 29966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.811062 29951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:03.811162 29951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "57be3fdb5e3848faafe57f62d941cbbf"
format_stamp: "Formatted at 2026-09-24 13:38:03 on dist-test-slave-m5z1"
server_key: "893a99f10a024a9915538a65c3f79237"
server_key_iv: "1cb9daf12ba60bb6588ff40767c759e0"
server_key_version: "encryptionkey@0"
I20260924 13:38:03.811242 29951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:03.830982 29951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:03.833317 29951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:03.833483 29951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:03.833735 29951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:03.834028 29951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:03.834081 29951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.834124 29951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:03.834172 29951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:03.840061 29951 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:40225
I20260924 13:38:03.840149 30079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:40225 every 8 connection(s)
I20260924 13:38:03.840440 29951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:03.848584 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 29951
I20260924 13:38:03.848690 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:03.848860 26561 external_mini_cluster.cc:1428] Setting key a310b3db202860b33f79a04fe9ddb81d
I20260924 13:38:03.849146 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:03.841116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:52297 (local address 127.25.240.126:40623)
0924 13:38:03.841176 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:03.841178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:03.841375 (+ 197us) server_negotiation.cc:407] Connection header received
0924 13:38:03.841916 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:03.841917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:03.841926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:03.841982 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:03.842850 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.843333 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.844253 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.844429 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.844478 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:03.846486 (+ 2008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:03.846487 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:03.846488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:03.846494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:03.847880 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.848207 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.848207 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.848208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.848259 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.848496 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.848497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.848497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.848592 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:03.848640 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:03.848972 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 13:38:03.849060 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:38:03.849392 30082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:03.841103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40623 (local address 127.25.240.67:52297)
0924 13:38:03.841254 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:03.841293 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:03.841829 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:03.842042 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:03.842045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:03.842587 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:03.842778 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:03.842782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.843408 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.843409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:03.844171 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:03.844173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.844499 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.844500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:03.844599 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:03.845013 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:03.845020 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:03.846354 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:03.847948 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:03.847953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:03.847954 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:03.848143 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:03.848316 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:03.848317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:03.848317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:03.848432 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:03.848655 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:03.848661 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:03.848822 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 13:38:03.848939 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 13:38:03.849113 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260924 13:38:03.849918 30080 heartbeater.cc:347] Connected to a master server at 127.25.240.126:40623
I20260924 13:38:03.850026 30080 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:03.850193 30080 heartbeater.cc:511] Master 127.25.240.126:40623 requested a full tablet report, sending...
I20260924 13:38:03.850586 29620 ts_manager.cc:194] Registered new tserver with Master: 57be3fdb5e3848faafe57f62d941cbbf (127.25.240.67:40225)
I20260924 13:38:03.850975 29620 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:52297
I20260924 13:38:03.860699 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[29576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257083, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:03.882357 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:03.875742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34502 (local address 127.25.240.126:40623)
0924 13:38:03.875806 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:03.875809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:03.875872 (+ 63us) server_negotiation.cc:407] Connection header received
0924 13:38:03.875918 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:03.875919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:03.875928 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:03.875981 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:03.876533 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.877009 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.877778 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:03.877958 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:03.878010 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:03.880183 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:03.880185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:03.880185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:03.880191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:03.881381 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.881735 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.881736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.881736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.881788 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:03.881932 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:03.881932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:03.881933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:03.882059 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:03.882126 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:03.882221 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 13:38:03.882284 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 13:38:03.882596 29620 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34502
I20260924 13:38:03.883024 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 29683
I20260924 13:38:03.887287 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 29817
I20260924 13:38:03.891428 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 29951
I20260924 13:38:03.895641 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 29589
2026-09-24T13:38:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1856 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[30092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[30092](info): set up 2 sockets
Sep 24 13:38:03 dist-test-slave-m5z1 krb5kdc[30092](info): commencing operation
krb5kdc: starting...
W20260924 13:38:04.925861 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:05.041003 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38011
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:38011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:05.119676 30105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:05.119870 30105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:05.119892 30105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:05.120951 30105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:05.120985 30105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:05.120999 30105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:05.121011 30105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:05.121023 30105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:05.122435 30105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:38011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:05.122612 30105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:05.122780 30105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:05.127393 30111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:05.127439 30110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:05.127559 30113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:05.127540 30105 server_base.cc:1034] running on GCE node
I20260924 13:38:05.127820 30105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:05.128137 30105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:05.129307 30105 hybrid_clock.cc:648] HybridClock initialized: now 1790257085129288 us; error 38 us; skew 500 ppm
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:05.131697 30105 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:05.132186 30105 webserver.cc:460] Webserver started at http://127.25.240.126:39003/ using document root <none> and password file <none>
I20260924 13:38:05.132429 30105 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:05.132498 30105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:05.132643 30105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:05.133747 30105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "e22439df64fc45aab7076480a8b0dd49"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "9cd7eaaa2e94d2b236019303de730c6a"
server_key_iv: "8453e0b4e1ecdeb6ba6c7d6d9c72f120"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.134117 30105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "e22439df64fc45aab7076480a8b0dd49"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "9cd7eaaa2e94d2b236019303de730c6a"
server_key_iv: "8453e0b4e1ecdeb6ba6c7d6d9c72f120"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.135424 30105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:38:05.136137 30120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.136310 30105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:05.136411 30105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "e22439df64fc45aab7076480a8b0dd49"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "9cd7eaaa2e94d2b236019303de730c6a"
server_key_iv: "8453e0b4e1ecdeb6ba6c7d6d9c72f120"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.136492 30105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:05.150349 30105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:05.152117 30105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:05.152281 30105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:05.156301 30105 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:38011
I20260924 13:38:05.156334 30172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:38011 every 8 connection(s)
I20260924 13:38:05.156823 30105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:05.157449 30173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:05.159636 30173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49: Bootstrap starting.
I20260924 13:38:05.160219 30173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:05.160523 30173 log.cc:824] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:05.161182 30173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49: No bootstrap required, opened a new log
I20260924 13:38:05.162736 30173 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22439df64fc45aab7076480a8b0dd49" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38011 } }
I20260924 13:38:05.162868 30173 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:05.162889 30173 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22439df64fc45aab7076480a8b0dd49, State: Initialized, Role: FOLLOWER
I20260924 13:38:05.163061 30173 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22439df64fc45aab7076480a8b0dd49" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38011 } }
I20260924 13:38:05.163122 30173 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:05.163143 30173 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:05.163209 30173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:05.164052 30173 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22439df64fc45aab7076480a8b0dd49" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38011 } }
I20260924 13:38:05.164161 30173 leader_election.cc:302] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22439df64fc45aab7076480a8b0dd49; no voters:
I20260924 13:38:05.164345 30173 leader_election.cc:288] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:05.164388 30176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:05.164556 30176 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [term 1 LEADER]: Becoming Leader. State: Replica: e22439df64fc45aab7076480a8b0dd49, State: Running, Role: LEADER
I20260924 13:38:05.164594 30173 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:05.164644 30176 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22439df64fc45aab7076480a8b0dd49" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38011 } }
I20260924 13:38:05.165046 30178 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22439df64fc45aab7076480a8b0dd49. Latest consensus state: current_term: 1 leader_uuid: "e22439df64fc45aab7076480a8b0dd49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22439df64fc45aab7076480a8b0dd49" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38011 } } }
I20260924 13:38:05.165182 30178 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:05.165316 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 30105
I20260924 13:38:05.165416 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:05.165038 30177 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22439df64fc45aab7076480a8b0dd49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22439df64fc45aab7076480a8b0dd49" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38011 } } }
I20260924 13:38:05.165580 30177 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:05.165644 26561 external_mini_cluster.cc:1428] Setting key b6fdc08004bef8981c2bb929f4592640
I20260924 13:38:05.165916 30185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:05.166437 30185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:05.167975 30185 catalog_manager.cc:1347] Generated new cluster ID: c395022a4ccf414ca831579e6915b2d4
I20260924 13:38:05.168036 30185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:05.175500 30185 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:05.176084 30185 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:05.177484 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.166171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.25.240.126:38011)
0924 13:38:05.166581 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:05.166590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:05.166628 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:38:05.166726 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:05.166731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:05.166772 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:05.166913 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:05.167635 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.169010 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.170080 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.170753 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.170870 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.173703 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:05.173707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:05.173707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:05.173719 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:05.175738 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.176151 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.176155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.176155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.176267 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.176501 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.176502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.176502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.176766 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:05.176909 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:05.177037 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 13:38:05.177281 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":68,"threads_started":1}
I20260924 13:38:05.185951 30185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e22439df64fc45aab7076480a8b0dd49: Generated new TSK 0
I20260924 13:38:05.186127 30185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:38:05.214708 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38011
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:05.294577 30199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:05.294780 30199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:05.294804 30199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:05.295965 30199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:05.296010 30199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:05.296036 30199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:05.297585 30199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:05.297832 30199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:05.298067 30199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:05.303185 30204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:05.303169 30205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:05.303169 30207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:05.303514 30199 server_base.cc:1034] running on GCE node
I20260924 13:38:05.303747 30199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:05.304021 30199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:05.305204 30199 hybrid_clock.cc:648] HybridClock initialized: now 1790257085305190 us; error 40 us; skew 500 ppm
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:05.307729 30199 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:38:05.308223 30199 webserver.cc:460] Webserver started at http://127.25.240.65:43555/ using document root <none> and password file <none>
I20260924 13:38:05.308461 30199 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:05.308528 30199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:05.308678 30199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:05.309764 30199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "a7b43940f45e4e3f94a8c386c4127fc4"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "d4e5eb319a6ec753ded17cf490bc8b2d"
server_key_iv: "b436303405cc502f74499df938cd4be5"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.310108 30199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "a7b43940f45e4e3f94a8c386c4127fc4"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "d4e5eb319a6ec753ded17cf490bc8b2d"
server_key_iv: "b436303405cc502f74499df938cd4be5"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.311388 30199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:05.312042 30214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.312227 30199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:05.312322 30199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "a7b43940f45e4e3f94a8c386c4127fc4"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "d4e5eb319a6ec753ded17cf490bc8b2d"
server_key_iv: "b436303405cc502f74499df938cd4be5"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.312399 30199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:05.329955 30199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:05.331930 30199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:05.332082 30199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:05.332295 30199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:05.332682 30199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:05.332734 30199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.332775 30199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:05.332803 30199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.339069 30199 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:34239
I20260924 13:38:05.339161 30327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:34239 every 8 connection(s)
I20260924 13:38:05.339457 30199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:05.348878 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.340205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:33345 (local address 127.25.240.126:38011)
0924 13:38:05.340249 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:05.340251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:05.340469 (+ 218us) server_negotiation.cc:407] Connection header received
0924 13:38:05.341430 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:05.341432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:05.341445 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:05.341496 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:05.342173 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.342693 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.343727 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.343963 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.344058 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.346123 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:05.346125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:05.346125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:05.346133 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:05.347440 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.347891 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.347893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.347893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.347950 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.348222 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.348223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.348224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.348343 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:05.348413 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:05.348694 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 13:38:05.348785 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:05.349062 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.340201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:38011 (local address 127.25.240.65:33345)
0924 13:38:05.340352 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:05.340386 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:05.341309 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:05.341605 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:05.341610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:05.341877 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:05.342066 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:05.342070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.342786 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.342787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:05.343633 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:05.343635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.344060 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.344061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:05.344193 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.344651 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:05.344658 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:05.346022 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:05.347563 (+ 1541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:05.347571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:05.347573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:05.347810 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:05.348027 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:05.348028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:05.348029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:05.348141 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:05.348417 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:05.348424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:05.348559 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 13:38:05.348661 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:38:05.348825 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260924 13:38:05.349694 30328 heartbeater.cc:347] Connected to a master server at 127.25.240.126:38011
I20260924 13:38:05.349745 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 30199
I20260924 13:38:05.349785 30328 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:05.349864 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:05.350001 30328 heartbeater.cc:511] Master 127.25.240.126:38011 requested a full tablet report, sending...
I20260924 13:38:05.350081 26561 external_mini_cluster.cc:1428] Setting key fecfc11bb044ed79f4fb56deba96a107
I20260924 13:38:05.350553 30137 ts_manager.cc:194] Registered new tserver with Master: a7b43940f45e4e3f94a8c386c4127fc4 (127.25.240.65:34239)
I20260924 13:38:05.351459 30137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:33345
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:38:05.376780 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38011
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:05.460345 30333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:05.460569 30333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:05.460608 30333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:05.461836 30333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:05.461894 30333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:05.461942 30333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:05.463389 30333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:05.463646 30333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:05.463868 30333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:05.468868 30341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:05.468865 30338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:05.468865 30339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:05.468955 30333 server_base.cc:1034] running on GCE node
I20260924 13:38:05.469259 30333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:05.469502 30333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:05.470666 30333 hybrid_clock.cc:648] HybridClock initialized: now 1790257085470648 us; error 30 us; skew 500 ppm
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:05.473187 30333 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:38:05.473699 30333 webserver.cc:460] Webserver started at http://127.25.240.66:33041/ using document root <none> and password file <none>
I20260924 13:38:05.473924 30333 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:05.473990 30333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:05.474125 30333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:05.475150 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "6e546ec926f54288aa0e5012b17605cb"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "52a3bcaa00931f14b89b4eda7f8b7560"
server_key_iv: "7da05c121d584a817e6f6afe3f8c1ce7"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.475518 30333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "6e546ec926f54288aa0e5012b17605cb"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "52a3bcaa00931f14b89b4eda7f8b7560"
server_key_iv: "7da05c121d584a817e6f6afe3f8c1ce7"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.476872 30333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:05.477548 30348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.477752 30333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:05.477840 30333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "6e546ec926f54288aa0e5012b17605cb"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "52a3bcaa00931f14b89b4eda7f8b7560"
server_key_iv: "7da05c121d584a817e6f6afe3f8c1ce7"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.477921 30333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:05.494656 30333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:05.496932 30333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:05.497092 30333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:05.497320 30333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:05.497674 30333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:05.497732 30333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.497772 30333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:05.497799 30333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.503727 30333 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:34101
I20260924 13:38:05.503780 30461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:34101 every 8 connection(s)
I20260924 13:38:05.504105 30333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:05.511624 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 30333
I20260924 13:38:05.511754 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:05.511924 26561 external_mini_cluster.cc:1428] Setting key 788996802ab9353e92b164f055a15f4a
I20260924 13:38:05.513324 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.504818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:48967 (local address 127.25.240.126:38011)
0924 13:38:05.504864 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:05.504866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:05.505072 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:38:05.506061 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:05.506063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:05.506075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:05.506139 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:05.506771 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.507279 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.508279 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.508476 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.508554 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.510577 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:05.510579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:05.510579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:05.510588 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:05.511849 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.512323 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.512324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.512325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.512389 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.512672 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.512673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.512673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.512792 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:05.512868 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:05.513138 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 13:38:05.513234 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:38:05.513478 30464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.504785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:38011 (local address 127.25.240.66:48967)
0924 13:38:05.504945 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:05.504988 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:05.505946 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:05.506211 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:05.506215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:05.506473 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:05.506654 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:05.506659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.507372 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.507373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:05.508181 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:05.508183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.508557 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.508558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:05.508692 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.509152 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:05.509160 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:05.510479 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:05.511926 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:05.511932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:05.511934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:05.512241 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:05.512462 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:05.512463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:05.512463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:05.512608 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:05.512868 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:05.512875 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:05.513006 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 13:38:05.513105 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 13:38:05.513255 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260924 13:38:05.513939 30462 heartbeater.cc:347] Connected to a master server at 127.25.240.126:38011
I20260924 13:38:05.514029 30462 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:05.514199 30462 heartbeater.cc:511] Master 127.25.240.126:38011 requested a full tablet report, sending...
I20260924 13:38:05.514540 30137 ts_manager.cc:194] Registered new tserver with Master: 6e546ec926f54288aa0e5012b17605cb (127.25.240.66:34101)
I20260924 13:38:05.514932 30137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:48967
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:38:05.536686 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38011
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:05.615777 30467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:05.615964 30467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:05.615986 30467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:05.617127 30467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:05.617170 30467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:05.617197 30467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:05.618680 30467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:38011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:05.618911 30467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:05.619138 30467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:05.624092 30473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:05.624121 30467 server_base.cc:1034] running on GCE node
W20260924 13:38:05.624119 30475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:05.624092 30472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:05.624495 30467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:05.624676 30467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:05.625813 30467 hybrid_clock.cc:648] HybridClock initialized: now 1790257085625794 us; error 40 us; skew 500 ppm
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:05.628188 30467 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:38:05.628634 30467 webserver.cc:460] Webserver started at http://127.25.240.67:41143/ using document root <none> and password file <none>
I20260924 13:38:05.628849 30467 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:05.628893 30467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:05.629050 30467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:05.630167 30467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "5e5bced8b17e4c45a656025c2fb99af9"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "ff57270c5a159a5d5ca55c31f77fa87e"
server_key_iv: "eca77e8ade81322dc7d6c68536759741"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.630519 30467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "5e5bced8b17e4c45a656025c2fb99af9"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "ff57270c5a159a5d5ca55c31f77fa87e"
server_key_iv: "eca77e8ade81322dc7d6c68536759741"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.631795 30467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:05.632463 30482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.632647 30467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:05.632742 30467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "5e5bced8b17e4c45a656025c2fb99af9"
format_stamp: "Formatted at 2026-09-24 13:38:05 on dist-test-slave-m5z1"
server_key: "ff57270c5a159a5d5ca55c31f77fa87e"
server_key_iv: "eca77e8ade81322dc7d6c68536759741"
server_key_version: "encryptionkey@0"
I20260924 13:38:05.632822 30467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:05.642062 30467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:05.643805 30467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:05.643958 30467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:05.644176 30467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:05.644484 30467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:05.644538 30467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.644579 30467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:05.644606 30467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:05.650357 30467 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:37659
I20260924 13:38:05.650398 30595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:37659 every 8 connection(s)
I20260924 13:38:05.650699 30467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:38:05.651212 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 30467
I20260924 13:38:05.651314 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:05.651463 26561 external_mini_cluster.cc:1428] Setting key d57d0d26703fb077768f761bdd558254
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:05.659865 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.651932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:36659 (local address 127.25.240.126:38011)
0924 13:38:05.651994 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:05.651996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:05.652260 (+ 264us) server_negotiation.cc:407] Connection header received
0924 13:38:05.653271 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:05.653272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:05.653281 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:05.653328 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:05.653904 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.654387 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.655344 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.655520 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.655569 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.657655 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:05.657656 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:05.657656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:05.657662 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:05.658803 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.659146 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.659146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.659147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.659192 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.659395 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.659395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.659395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.659487 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:05.659535 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:05.659748 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 13:38:05.659804 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:38:05.660032 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.651925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:38011 (local address 127.25.240.67:36659)
0924 13:38:05.652114 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:05.652161 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:05.653196 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:05.653386 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:05.653389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:05.653645 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:05.653833 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:05.653837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.654471 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.654472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:05.655270 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:05.655272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.655588 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.655589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:05.655684 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.656205 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:05.656214 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:05.657519 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:05.658872 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:05.658877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:05.658879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:05.659081 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:05.659249 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:05.659250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:05.659250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:05.659342 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:05.659539 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:05.659543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:05.659644 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 13:38:05.659720 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:38:05.659838 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260924 13:38:05.660498 30596 heartbeater.cc:347] Connected to a master server at 127.25.240.126:38011
I20260924 13:38:05.660605 30596 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:05.660784 30596 heartbeater.cc:511] Master 127.25.240.126:38011 requested a full tablet report, sending...
I20260924 13:38:05.661128 30137 ts_manager.cc:194] Registered new tserver with Master: 5e5bced8b17e4c45a656025c2fb99af9 (127.25.240.67:37659)
I20260924 13:38:05.661449 30137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:36659
I20260924 13:38:05.663319 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:38:05.685756 30603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.678214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60946 (local address 127.25.240.65:34239)
0924 13:38:05.678382 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:05.678387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:05.678397 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:05.678422 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:05.678424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:05.678437 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:05.678501 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:05.679000 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.679685 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.680403 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.680802 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.680865 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.682942 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:05.682950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:05.682950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:05.682956 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:05.684496 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.684887 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.684890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.684891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.684951 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.685168 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.685169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.685169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.685310 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:05.685463 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:05.685535 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 13:38:05.685642 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:05.693163 30195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:05.687009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.25.240.126:38011)
0924 13:38:05.687061 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:05.687063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:05.687071 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:05.687137 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:05.687137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:05.687145 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:05.687184 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:05.687616 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.688073 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.688692 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:05.688872 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:05.688919 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:05.690875 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:05.690876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:05.690877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:05.690882 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:05.692235 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.692510 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.692510 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.692511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.692570 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:05.692714 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:05.692715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:05.692715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:05.692846 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:05.692913 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:05.692996 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 13:38:05.693054 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 13:38:05.693401 30137 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59850
I20260924 13:38:05.693737 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 30199
I20260924 13:38:05.699131 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 30333
I20260924 13:38:05.704167 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 30467
I20260924 13:38:05.708690 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 30105
2026-09-24T13:38:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1812 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30608](info): set up 2 sockets
Sep 24 13:38:05 dist-test-slave-m5z1 krb5kdc[30608](info): commencing operation
krb5kdc: starting...
W20260924 13:38:06.761878 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:06 dist-test-slave-m5z1 krb5kdc[30608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:06.878161 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:34341
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:06.957641 30621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:06.957839 30621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:06.957862 30621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:06.959023 30621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:06.959066 30621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:06.959080 30621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:06.959093 30621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:06.959105 30621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:06.960631 30621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:06.960872 30621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:06.961104 30621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:06.965930 30629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:06.965931 30626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:06.965965 30621 server_base.cc:1034] running on GCE node
W20260924 13:38:06.965931 30627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:06.966408 30621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:06.966694 30621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:06.967857 30621 hybrid_clock.cc:648] HybridClock initialized: now 1790257086967839 us; error 30 us; skew 500 ppm
Sep 24 13:38:06 dist-test-slave-m5z1 krb5kdc[30608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257086, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:06.970402 30621 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:06.970841 30621 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:06.970914 30621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:06.971058 30621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:06.972090 30621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "3e6750d0eeb4454280bce39ba99b1b80"
format_stamp: "Formatted at 2026-09-24 13:38:06 on dist-test-slave-m5z1"
server_key: "de5624c04b9d4358ee6cb4d351cd4f31"
server_key_iv: "c1df81cd2ea8161a39246dd0df4089f1"
server_key_version: "encryptionkey@0"
I20260924 13:38:06.972452 30621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "3e6750d0eeb4454280bce39ba99b1b80"
format_stamp: "Formatted at 2026-09-24 13:38:06 on dist-test-slave-m5z1"
server_key: "de5624c04b9d4358ee6cb4d351cd4f31"
server_key_iv: "c1df81cd2ea8161a39246dd0df4089f1"
server_key_version: "encryptionkey@0"
I20260924 13:38:06.973826 30621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:38:06.974575 30635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:06.974843 30621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:06.974915 30621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "3e6750d0eeb4454280bce39ba99b1b80"
format_stamp: "Formatted at 2026-09-24 13:38:06 on dist-test-slave-m5z1"
server_key: "de5624c04b9d4358ee6cb4d351cd4f31"
server_key_iv: "c1df81cd2ea8161a39246dd0df4089f1"
server_key_version: "encryptionkey@0"
I20260924 13:38:06.974993 30621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:06.990309 30621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:06.992105 30621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:06.992271 30621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:06.996292 30621 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:34341
I20260924 13:38:06.996323 30687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:34341 every 8 connection(s)
I20260924 13:38:06.996606 30621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:06.997484 30688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:06.999702 30688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80: Bootstrap starting.
I20260924 13:38:07.000244 30688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:07.000478 30688 log.cc:824] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:07.001116 30688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80: No bootstrap required, opened a new log
I20260924 13:38:07.002833 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 30621
I20260924 13:38:07.002802 30688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6750d0eeb4454280bce39ba99b1b80" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34341 } }
I20260924 13:38:07.002946 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:07.002969 30688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:07.003010 30688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6750d0eeb4454280bce39ba99b1b80, State: Initialized, Role: FOLLOWER
I20260924 13:38:07.003157 26561 external_mini_cluster.cc:1428] Setting key f47c0eea61b76972c4469ef97be7651b
I20260924 13:38:07.003154 30688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6750d0eeb4454280bce39ba99b1b80" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34341 } }
I20260924 13:38:07.003238 30688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:07.003286 30688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:07.003336 30688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:07.003983 30688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6750d0eeb4454280bce39ba99b1b80" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34341 } }
I20260924 13:38:07.004112 30688 leader_election.cc:302] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e6750d0eeb4454280bce39ba99b1b80; no voters:
I20260924 13:38:07.004299 30688 leader_election.cc:288] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:07.004364 30693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:07.004521 30693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [term 1 LEADER]: Becoming Leader. State: Replica: 3e6750d0eeb4454280bce39ba99b1b80, State: Running, Role: LEADER
I20260924 13:38:07.004557 30688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:07.004606 30693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6750d0eeb4454280bce39ba99b1b80" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34341 } }
I20260924 13:38:07.005033 30694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e6750d0eeb4454280bce39ba99b1b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6750d0eeb4454280bce39ba99b1b80" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34341 } } }
I20260924 13:38:07.005113 30694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:07.005534 30695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e6750d0eeb4454280bce39ba99b1b80. Latest consensus state: current_term: 1 leader_uuid: "3e6750d0eeb4454280bce39ba99b1b80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e6750d0eeb4454280bce39ba99b1b80" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34341 } } }
I20260924 13:38:07.005676 30695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:07.005815 30701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:07.006287 30701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:07.007746 30701 catalog_manager.cc:1347] Generated new cluster ID: c5a34507979d49e29bacc07baff45833
I20260924 13:38:07.007800 30701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:38:07 dist-test-slave-m5z1 krb5kdc[30608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:07.014376 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.003644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.25.240.126:34341)
0924 13:38:07.004265 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.004275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.004311 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:38:07.004435 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.004442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.004494 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.005500 (+ 1006us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:07.005634 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.007124 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.008056 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.008494 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.008623 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.011013 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:07.011017 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:07.011018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:07.011037 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:07.012744 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.013123 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.013127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.013129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.013232 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.013459 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.013460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.013460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.013758 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:07.013945 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.014013 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.014153 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":498,"threads_started":1}
I20260924 13:38:07.033651 30701 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:07.034217 30701 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:07.040444 30701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80: Generated new TSK 0
I20260924 13:38:07.040632 30701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:38:07.072209 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34341
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:07.152374 30714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:07.152587 30714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:07.152613 30714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:07.153734 30714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:07.153774 30714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:07.153800 30714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:07.155227 30714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:07.155405 30714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:07.155573 30714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:07.160507 30719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.160529 30722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.160532 30720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:07.160563 30714 server_base.cc:1034] running on GCE node
I20260924 13:38:07.160941 30714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:07.161129 30714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:07.162268 30714 hybrid_clock.cc:648] HybridClock initialized: now 1790257087162250 us; error 33 us; skew 500 ppm
Sep 24 13:38:07 dist-test-slave-m5z1 krb5kdc[30608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257087, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:07.167477 30714 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:38:07.167829 30714 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:07.167902 30714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:07.168049 30714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:07.169085 30714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "691d6682091a4e1b9ba6c4a22e381418"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "b8d882053da014ae670df99b235e315f"
server_key_iv: "b4493277b38cd3fc8926410b52b1dc1d"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.169452 30714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "691d6682091a4e1b9ba6c4a22e381418"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "b8d882053da014ae670df99b235e315f"
server_key_iv: "b4493277b38cd3fc8926410b52b1dc1d"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.170734 30714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:07.171434 30728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.171592 30714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:07.171684 30714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "691d6682091a4e1b9ba6c4a22e381418"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "b8d882053da014ae670df99b235e315f"
server_key_iv: "b4493277b38cd3fc8926410b52b1dc1d"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.171770 30714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:07.184114 30714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:07.186447 30714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:07.186614 30714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:07.186843 30714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:07.187152 30714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:07.187211 30714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.187244 30714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:07.187270 30714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.192974 30714 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:40775
I20260924 13:38:07.193032 30841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:40775 every 8 connection(s)
I20260924 13:38:07.193271 30714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:38:07.197032 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 30714
I20260924 13:38:07.197175 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:07.197346 26561 external_mini_cluster.cc:1428] Setting key 92f2a82f178a3e844d27d3b109741b75
Sep 24 13:38:07 dist-test-slave-m5z1 krb5kdc[30608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257087, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:07.202976 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.194008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:59489 (local address 127.25.240.126:34341)
0924 13:38:07.194087 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.194089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.194277 (+ 188us) server_negotiation.cc:407] Connection header received
0924 13:38:07.194791 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.194792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.194809 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.194872 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:07.195777 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.196293 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.197349 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.197688 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.197761 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.200194 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:07.200196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:07.200196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:07.200202 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:07.201623 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.202021 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.202022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.202023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.202085 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.202336 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.202337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.202337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.202453 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:07.202523 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.202806 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.202892 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:38:07.203207 30844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.193960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:34341 (local address 127.25.240.65:59489)
0924 13:38:07.194145 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.194187 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:07.194680 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:07.194940 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:07.194944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:07.195193 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:07.195663 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.195667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.196389 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.196389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.197269 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.197272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.197770 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.197771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.197927 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.198460 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:07.198468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:07.200068 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:07.201703 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:07.201710 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:07.201712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:07.201930 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:07.202151 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:07.202152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:07.202152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:07.202264 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:07.202525 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:07.202532 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:07.202673 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 13:38:07.202773 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 13:38:07.202923 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260924 13:38:07.203703 30842 heartbeater.cc:347] Connected to a master server at 127.25.240.126:34341
I20260924 13:38:07.203794 30842 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:07.203958 30842 heartbeater.cc:511] Master 127.25.240.126:34341 requested a full tablet report, sending...
I20260924 13:38:07.204449 30652 ts_manager.cc:194] Registered new tserver with Master: 691d6682091a4e1b9ba6c4a22e381418 (127.25.240.65:40775)
I20260924 13:38:07.205132 30652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:59489
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:38:07.222514 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34341
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:07.302987 30847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:07.303184 30847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:07.303208 30847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:07.304271 30847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:07.304303 30847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:07.304329 30847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:07.305971 30847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:07.306159 30847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:07.306368 30847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:07.311089 30855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.311079 30852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.311072 30853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:07.311301 30847 server_base.cc:1034] running on GCE node
I20260924 13:38:07.311548 30847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:07.311736 30847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:07.312870 30847 hybrid_clock.cc:648] HybridClock initialized: now 1790257087312859 us; error 26 us; skew 500 ppm
Sep 24 13:38:07 dist-test-slave-m5z1 krb5kdc[30608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257087, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:07.315302 30847 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:38:07.315639 30847 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:07.315686 30847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:07.315774 30847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:07.316752 30847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "362cc05e475a46e38735bc4082b5b8ac"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "3dff3be4479406c0bc40f67947d91546"
server_key_iv: "31f342e2b2de845bfa9e1e97345da8d0"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.317044 30847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "362cc05e475a46e38735bc4082b5b8ac"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "3dff3be4479406c0bc40f67947d91546"
server_key_iv: "31f342e2b2de845bfa9e1e97345da8d0"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.318351 30847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:07.319015 30861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.319183 30847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:07.319262 30847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "362cc05e475a46e38735bc4082b5b8ac"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "3dff3be4479406c0bc40f67947d91546"
server_key_iv: "31f342e2b2de845bfa9e1e97345da8d0"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.319320 30847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:07.333168 30847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:07.335058 30847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:07.335166 30847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:07.335342 30847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:07.335654 30847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:07.335688 30847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.335711 30847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:07.335759 30847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.341462 30847 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:44299
I20260924 13:38:07.341532 30974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:44299 every 8 connection(s)
I20260924 13:38:07.341820 30847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:38:07.346841 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 30847
I20260924 13:38:07.346920 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:07.347105 26561 external_mini_cluster.cc:1428] Setting key 17d511ce6dbe2cea966adc536df33f6c
Sep 24 13:38:07 dist-test-slave-m5z1 krb5kdc[30608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257087, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:07.351609 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.342548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:45863 (local address 127.25.240.126:34341)
0924 13:38:07.342614 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.342616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.342804 (+ 188us) server_negotiation.cc:407] Connection header received
0924 13:38:07.343281 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.343282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.343291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.343352 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:07.344228 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.344718 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.345747 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.345962 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.346055 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.348494 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:07.348496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:07.348496 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:07.348504 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:07.350222 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.350644 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.350646 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.350646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.350708 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.350965 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.350966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.350967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.351086 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:07.351159 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.351441 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.351520 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:38:07.351790 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.342515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:34341 (local address 127.25.240.66:45863)
0924 13:38:07.342680 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.342721 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:07.343198 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:07.343402 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:07.343405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:07.343658 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:07.344153 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.344158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.344795 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.344795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.345652 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.345655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.346097 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.346098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.346256 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.346784 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:07.346792 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:07.348396 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:07.350323 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:07.350330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:07.350332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:07.350549 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:07.350783 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:07.350783 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:07.350784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:07.350894 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:07.351160 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:07.351167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:07.351299 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 13:38:07.351407 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 13:38:07.351555 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260924 13:38:07.352397 30975 heartbeater.cc:347] Connected to a master server at 127.25.240.126:34341
I20260924 13:38:07.352473 30975 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:07.352653 30975 heartbeater.cc:511] Master 127.25.240.126:34341 requested a full tablet report, sending...
I20260924 13:38:07.353041 30652 ts_manager.cc:194] Registered new tserver with Master: 362cc05e475a46e38735bc4082b5b8ac (127.25.240.66:44299)
I20260924 13:38:07.353631 30652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:45863
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:38:07.372815 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34341
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:07.453493 30980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:07.453713 30980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:07.453737 30980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:07.454821 30980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:07.454856 30980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:07.454882 30980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:07.456284 30980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:07.456467 30980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:07.456629 30980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:38:07.461483 30980 server_base.cc:1034] running on GCE node
W20260924 13:38:07.461501 30988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.461489 30985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.461684 30986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:07.461897 30980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:07.462141 30980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:07.463299 30980 hybrid_clock.cc:648] HybridClock initialized: now 1790257087463285 us; error 45 us; skew 500 ppm
Sep 24 13:38:07 dist-test-slave-m5z1 krb5kdc[30608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257087, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:07.465776 30980 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:38:07.466077 30980 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:07.466120 30980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:07.466209 30980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:07.467167 30980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "a5b89c24e7e1471da36354920597b18d"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "6a06abdf39fadefda4080842e5e550cf"
server_key_iv: "d1c6393d741b8c8fbf3bbfe55e7090d3"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.467473 30980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "a5b89c24e7e1471da36354920597b18d"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "6a06abdf39fadefda4080842e5e550cf"
server_key_iv: "d1c6393d741b8c8fbf3bbfe55e7090d3"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.468655 30980 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:38:07.469301 30994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.469455 30980 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:07.469535 30980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "a5b89c24e7e1471da36354920597b18d"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "6a06abdf39fadefda4080842e5e550cf"
server_key_iv: "d1c6393d741b8c8fbf3bbfe55e7090d3"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.469635 30980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:07.489641 30980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:07.491952 30980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:07.492067 30980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:07.492251 30980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:07.492507 30980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:07.492537 30980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.492559 30980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:07.492584 30980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.498075 30980 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:33621
I20260924 13:38:07.498150 31107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:33621 every 8 connection(s)
I20260924 13:38:07.498415 30980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:38:07 dist-test-slave-m5z1 krb5kdc[30608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257087, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:07.507416 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.499133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:39359 (local address 127.25.240.126:34341)
0924 13:38:07.499200 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.499202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.499401 (+ 199us) server_negotiation.cc:407] Connection header received
0924 13:38:07.499846 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.499847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.499858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.499909 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:07.500767 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.501321 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.502379 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.502564 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.502621 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.504794 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:07.504796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:07.504796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:07.504806 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:07.506283 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.506643 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.506644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.506644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.506694 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.506917 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.506917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.506917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.507015 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:07.507064 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.507294 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.507352 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 13:38:07.507633 31110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.499088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:34341 (local address 127.25.240.67:39359)
0924 13:38:07.499272 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.499307 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:07.499754 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:07.499960 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:07.499964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:07.500213 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:07.500691 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.500696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.501411 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.501411 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.502296 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.502299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.502638 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.502639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.502736 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.503278 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:07.503287 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:07.504645 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:07.506361 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:07.506367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:07.506368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:07.506579 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:07.506751 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:07.506752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:07.506752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:07.506862 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:07.507074 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:07.507078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:07.507184 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 13:38:07.507264 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:38:07.507377 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":59,"threads_started":1}
I20260924 13:38:07.507835 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 30980
I20260924 13:38:07.507939 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:07.508144 26561 external_mini_cluster.cc:1428] Setting key 402c81f513d0f4d78e222268cfcf7ae5
I20260924 13:38:07.508160 31108 heartbeater.cc:347] Connected to a master server at 127.25.240.126:34341
I20260924 13:38:07.508235 31108 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:07.508395 31108 heartbeater.cc:511] Master 127.25.240.126:34341 requested a full tablet report, sending...
I20260924 13:38:07.508801 30652 ts_manager.cc:194] Registered new tserver with Master: a5b89c24e7e1471da36354920597b18d (127.25.240.67:33621)
I20260924 13:38:07.509056 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:38:07.509253 30652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:39359
I20260924 13:38:07.516230 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.510541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.25.240.126:34341)
0924 13:38:07.510595 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.510597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.510705 (+ 108us) server_negotiation.cc:407] Connection header received
0924 13:38:07.510773 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.510774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.510783 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.510838 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:07.511398 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.511856 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.512598 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.512799 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.512871 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.513981 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:07.513983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:07.513983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:07.513989 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:07.515105 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.515458 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.515459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.515459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.515514 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:07.515708 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:07.515709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:07.515709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:07.515852 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:07.515953 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.516066 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.516136 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:38:07.517542 30652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:38:07.519179 30652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:07.525625 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.521283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40775 (local address 127.0.0.1:35658)
0924 13:38:07.521390 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.521423 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:07.521487 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:07.522144 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:07.522145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:07.522147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:07.522195 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.522196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.523309 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.523310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.524387 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.524389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.525333 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.525335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.525406 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.525410 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:07.525440 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 13:38:07.525500 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":44,"thread_start_us":53,"threads_started":1}
I20260924 13:38:07.525725 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.521245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:33621 (local address 127.0.0.1:58720)
0924 13:38:07.521383 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.521422 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:07.521485 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:07.521849 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:07.521852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:07.521855 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:07.521921 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.521922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.523222 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.523224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.524580 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.524583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.525421 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.525421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.525492 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.525495 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:07.525546 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:38:07.525613 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":84,"thread_start_us":55,"threads_started":1}
I20260924 13:38:07.525803 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.521628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.25.240.66:44299)
0924 13:38:07.521838 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.521842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.522051 (+ 209us) server_negotiation.cc:407] Connection header received
0924 13:38:07.522090 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.522092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.522106 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.522182 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:07.522417 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.523468 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.524729 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.525203 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.525292 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.525351 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.525668 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.525710 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":88,"threads_started":1}
I20260924 13:38:07.525627 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.521437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:44299 (local address 127.0.0.1:33552)
0924 13:38:07.521911 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.521922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:07.521984 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:07.522279 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:07.522280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:07.522283 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:07.522344 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.522345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.523561 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.523562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.524662 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.524664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.525291 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.525292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.525379 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.525392 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:38:07.525460 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 13:38:07.525508 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":3,"thread_start_us":38,"threads_started":1}
I20260924 13:38:07.526026 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.521404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.25.240.67:33621)
0924 13:38:07.521685 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.521690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.521702 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:38:07.521739 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.521741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.521756 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.521839 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:07.521989 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.523128 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.524669 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.525326 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.525413 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.525470 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.525589 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.525628 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":83,"threads_started":1}
I20260924 13:38:07.526594 30909 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc00e9e4d3d41d69699e9320ed11664 (DEFAULT_TABLE table=test-table [id=ebc67ce231bf4abab4f632add2b55a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:07.526643 31042 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc00e9e4d3d41d69699e9320ed11664 (DEFAULT_TABLE table=test-table [id=ebc67ce231bf4abab4f632add2b55a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:07.526921 30909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc00e9e4d3d41d69699e9320ed11664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:07.526932 31042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc00e9e4d3d41d69699e9320ed11664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:07.529405 31122 tablet_bootstrap.cc:492] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: Bootstrap starting.
I20260924 13:38:07.529527 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.521623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.25.240.65:40775)
0924 13:38:07.521794 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.521798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.521809 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:38:07.521849 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.521851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.521866 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.522269 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:07.522357 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.523152 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.524476 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.525227 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.525975 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.526044 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.526070 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.529445 (+ 3375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260924 13:38:07.529654 31123 tablet_bootstrap.cc:492] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Bootstrap starting.
I20260924 13:38:07.530082 30776 tablet_service.cc:1467] Processing CreateTablet for tablet 2cc00e9e4d3d41d69699e9320ed11664 (DEFAULT_TABLE table=test-table [id=ebc67ce231bf4abab4f632add2b55a34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:07.530274 31123 tablet_bootstrap.cc:654] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:07.530354 31122 tablet_bootstrap.cc:654] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:07.530400 30776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cc00e9e4d3d41d69699e9320ed11664. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:07.530956 31122 log.cc:824] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:07.530956 31123 log.cc:824] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:07.531711 31123 tablet_bootstrap.cc:492] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: No bootstrap required, opened a new log
I20260924 13:38:07.531781 31123 ts_tablet_manager.cc:1397] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:07.531955 31122 tablet_bootstrap.cc:492] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: No bootstrap required, opened a new log
I20260924 13:38:07.532016 31122 ts_tablet_manager.cc:1397] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 13:38:07.533285 31126 tablet_bootstrap.cc:492] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418: Bootstrap starting.
I20260924 13:38:07.533834 31123 raft_consensus.cc:348] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.533964 31126 tablet_bootstrap.cc:654] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:07.534008 31123 raft_consensus.cc:374] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:07.534049 31123 raft_consensus.cc:729] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 362cc05e475a46e38735bc4082b5b8ac, State: Initialized, Role: FOLLOWER
I20260924 13:38:07.534027 31122 raft_consensus.cc:348] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.534165 31122 raft_consensus.cc:374] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:07.534200 31122 raft_consensus.cc:729] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5b89c24e7e1471da36354920597b18d, State: Initialized, Role: FOLLOWER
I20260924 13:38:07.534201 31123 consensus_queue.cc:260] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.534356 31122 consensus_queue.cc:260] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.534495 31123 ts_tablet_manager.cc:1428] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 13:38:07.534550 30975 heartbeater.cc:503] Master 127.25.240.126:34341 was elected leader, sending a full tablet report...
I20260924 13:38:07.534574 31126 log.cc:824] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:07.534615 31122 ts_tablet_manager.cc:1428] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:38:07.534737 31108 heartbeater.cc:503] Master 127.25.240.126:34341 was elected leader, sending a full tablet report...
I20260924 13:38:07.535343 31126 tablet_bootstrap.cc:492] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418: No bootstrap required, opened a new log
I20260924 13:38:07.535447 31126 ts_tablet_manager.cc:1397] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:07.536919 31126 raft_consensus.cc:348] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.537042 31126 raft_consensus.cc:374] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:07.537065 31126 raft_consensus.cc:729] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691d6682091a4e1b9ba6c4a22e381418, State: Initialized, Role: FOLLOWER
I20260924 13:38:07.537140 31126 consensus_queue.cc:260] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.537384 31126 ts_tablet_manager.cc:1428] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:07.537392 30842 heartbeater.cc:503] Master 127.25.240.126:34341 was elected leader, sending a full tablet report...
W20260924 13:38:07.592417 30976 tablet.cc:2369] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:38:07.684792 31127 raft_consensus.cc:482] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:07.684926 31127 raft_consensus.cc:504] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.685266 31127 leader_election.cc:288] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5b89c24e7e1471da36354920597b18d (127.25.240.67:33621), 691d6682091a4e1b9ba6c4a22e381418 (127.25.240.65:40775)
I20260924 13:38:07.688202 31131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.685470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40775 (local address 127.25.240.66:45931)
0924 13:38:07.685782 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.685793 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:07.685851 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:07.685998 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:07.685999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:07.686000 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:07.686061 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.686062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.686696 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.686697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.687440 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.687441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.687989 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.687990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.688061 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.688064 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:07.688113 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:38:07.688140 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":88,"thread_start_us":127,"threads_started":1}
I20260924 13:38:07.688303 31120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.685525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:45931 (local address 127.25.240.65:40775)
0924 13:38:07.685604 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.685606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.685865 (+ 259us) server_negotiation.cc:407] Connection header received
0924 13:38:07.685897 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.685899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.685908 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.685972 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:07.686132 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.686614 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.687538 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.687901 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.687957 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.687964 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.688155 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.688213 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:38:07.688369 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.685463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:33621 (local address 127.25.240.66:45763)
0924 13:38:07.685545 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.685580 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:07.685640 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:07.685825 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:07.685825 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:07.685827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:07.685919 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.685921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.686603 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.686604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.687439 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:07.687442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.688050 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.688051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:07.688179 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.688182 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:07.688258 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:38:07.688293 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":26}
I20260924 13:38:07.688401 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.685522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:45763 (local address 127.25.240.67:33621)
0924 13:38:07.685584 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.685586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.685623 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:38:07.685730 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.685731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.685741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.685826 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:07.685997 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.686520 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.687536 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.687953 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.688027 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.688035 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.688276 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.688319 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:38:07.688591 30796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc00e9e4d3d41d69699e9320ed11664" candidate_uuid: "362cc05e475a46e38735bc4082b5b8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691d6682091a4e1b9ba6c4a22e381418" is_pre_election: true
I20260924 13:38:07.688683 31062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc00e9e4d3d41d69699e9320ed11664" candidate_uuid: "362cc05e475a46e38735bc4082b5b8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b89c24e7e1471da36354920597b18d" is_pre_election: true
I20260924 13:38:07.688748 30796 raft_consensus.cc:2454] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362cc05e475a46e38735bc4082b5b8ac in term 0.
I20260924 13:38:07.688879 31062 raft_consensus.cc:2454] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 362cc05e475a46e38735bc4082b5b8ac in term 0.
I20260924 13:38:07.689047 30864 leader_election.cc:302] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362cc05e475a46e38735bc4082b5b8ac, 691d6682091a4e1b9ba6c4a22e381418; no voters:
I20260924 13:38:07.689186 31127 raft_consensus.cc:2781] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:07.689257 31127 raft_consensus.cc:482] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:07.689276 31127 raft_consensus.cc:3037] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:07.689927 31127 raft_consensus.cc:504] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.690066 31127 leader_election.cc:288] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [CANDIDATE]: Term 1 election: Requested vote from peers a5b89c24e7e1471da36354920597b18d (127.25.240.67:33621), 691d6682091a4e1b9ba6c4a22e381418 (127.25.240.65:40775)
I20260924 13:38:07.690189 31062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc00e9e4d3d41d69699e9320ed11664" candidate_uuid: "362cc05e475a46e38735bc4082b5b8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5b89c24e7e1471da36354920597b18d"
I20260924 13:38:07.690212 30796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cc00e9e4d3d41d69699e9320ed11664" candidate_uuid: "362cc05e475a46e38735bc4082b5b8ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "691d6682091a4e1b9ba6c4a22e381418"
I20260924 13:38:07.690275 31062 raft_consensus.cc:3037] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:07.690287 30796 raft_consensus.cc:3037] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:07.690950 31062 raft_consensus.cc:2454] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362cc05e475a46e38735bc4082b5b8ac in term 1.
I20260924 13:38:07.691022 30796 raft_consensus.cc:2454] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 362cc05e475a46e38735bc4082b5b8ac in term 1.
I20260924 13:38:07.691180 30863 leader_election.cc:302] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 362cc05e475a46e38735bc4082b5b8ac, a5b89c24e7e1471da36354920597b18d; no voters:
I20260924 13:38:07.691303 31127 raft_consensus.cc:2781] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:07.691488 31127 raft_consensus.cc:686] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 1 LEADER]: Becoming Leader. State: Replica: 362cc05e475a46e38735bc4082b5b8ac, State: Running, Role: LEADER
I20260924 13:38:07.691601 31127 consensus_queue.cc:237] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } }
I20260924 13:38:07.692466 30651 catalog_manager.cc:5495] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 362cc05e475a46e38735bc4082b5b8ac (127.25.240.66). New cstate: current_term: 1 leader_uuid: "362cc05e475a46e38735bc4082b5b8ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "362cc05e475a46e38735bc4082b5b8ac" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44299 } health_report { overall_health: HEALTHY } } }
W20260924 13:38:07.693944 30843 tablet.cc:2369] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:38:07.706853 31121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.703971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33554 (local address 127.25.240.66:44299)
0924 13:38:07.704009 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.704011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.704020 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:07.704083 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.704084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.704097 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.704152 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:07.704260 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.704777 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.705529 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:07.706034 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:07.706115 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:07.706363 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:07.706509 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:07.706612 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.706676 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.706697 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:38:07.709182 31062 raft_consensus.cc:1261] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 1 FOLLOWER]: Refusing update from remote peer 362cc05e475a46e38735bc4082b5b8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:07.709245 30796 raft_consensus.cc:1261] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 1 FOLLOWER]: Refusing update from remote peer 362cc05e475a46e38735bc4082b5b8ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:07.709545 31127 consensus_queue.cc:1035] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5b89c24e7e1471da36354920597b18d" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 33621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:07.709725 31132 consensus_queue.cc:1035] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "691d6682091a4e1b9ba6c4a22e381418" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:07.711901 31136 mvcc.cc:205] Tried to move back new op lower bound from 7332893031253610496 to 7332893031185403904. Current Snapshot: MvccSnapshot[applied={T|T < 7332893031253610496}]
I20260924 13:38:07.713810 30651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:07.713907 30651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:07.714680 30651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e6750d0eeb4454280bce39ba99b1b80: Sending DeleteTablet for 3 replicas of tablet 2cc00e9e4d3d41d69699e9320ed11664
I20260924 13:38:07.715016 30776 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc00e9e4d3d41d69699e9320ed11664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:07 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:35658
I20260924 13:38:07.715263 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 30714
I20260924 13:38:07.715261 31144 tablet_replica.cc:330] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418: stopping tablet replica
I20260924 13:38:07.715252 31042 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc00e9e4d3d41d69699e9320ed11664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:07 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:58720
I20260924 13:38:07.715358 31144 raft_consensus.cc:2229] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:07.715461 31144 raft_consensus.cc:2258] T 2cc00e9e4d3d41d69699e9320ed11664 P 691d6682091a4e1b9ba6c4a22e381418 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:07.716151 30909 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cc00e9e4d3d41d69699e9320ed11664 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:07 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:33552
I20260924 13:38:07.716214 31145 tablet_replica.cc:330] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: stopping tablet replica
I20260924 13:38:07.716305 31145 raft_consensus.cc:2229] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:07.716341 31146 tablet_replica.cc:330] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: stopping tablet replica
I20260924 13:38:07.716395 31145 raft_consensus.cc:2258] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:07.716418 31146 raft_consensus.cc:2229] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:38:07.716495 31146 raft_consensus.cc:2258] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:07.716872 31145 ts_tablet_manager.cc:1905] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:07.716915 31146 ts_tablet_manager.cc:1905] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:07.718674 31146 ts_tablet_manager.cc:1918] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:07.718743 31146 log.cc:1196] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/2cc00e9e4d3d41d69699e9320ed11664
I20260924 13:38:07.718732 31145 ts_tablet_manager.cc:1918] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:07.718792 31145 log.cc:1196] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/2cc00e9e4d3d41d69699e9320ed11664
I20260924 13:38:07.719017 31146 ts_tablet_manager.cc:1939] T 2cc00e9e4d3d41d69699e9320ed11664 P 362cc05e475a46e38735bc4082b5b8ac: Deleting consensus metadata
I20260924 13:38:07.719070 31145 ts_tablet_manager.cc:1939] T 2cc00e9e4d3d41d69699e9320ed11664 P a5b89c24e7e1471da36354920597b18d: Deleting consensus metadata
I20260924 13:38:07.719417 30637 catalog_manager.cc:4841] TS 362cc05e475a46e38735bc4082b5b8ac (127.25.240.66:44299): tablet 2cc00e9e4d3d41d69699e9320ed11664 (table test-table [id=ebc67ce231bf4abab4f632add2b55a34]) successfully deleted
I20260924 13:38:07.719583 30637 catalog_manager.cc:4841] TS a5b89c24e7e1471da36354920597b18d (127.25.240.67:33621): tablet 2cc00e9e4d3d41d69699e9320ed11664 (table test-table [id=ebc67ce231bf4abab4f632add2b55a34]) successfully deleted
W20260924 13:38:07.722527 30638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:40775 (error 108)
I20260924 13:38:07.722725 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 30847
I20260924 13:38:07.723297 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.722869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40775 (local address 127.0.0.1:35670)
0924 13:38:07.723221 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:07.723259 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:40775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332}
W20260924 13:38:07.723467 30638 catalog_manager.cc:4584] TS 691d6682091a4e1b9ba6c4a22e381418 (127.25.240.65:40775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cc00e9e4d3d41d69699e9320ed11664: Network error: Client connection negotiation failed: client connection to 127.25.240.65:40775: connect: Connection refused (error 111)
I20260924 13:38:07.727440 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 30980
I20260924 13:38:07.731721 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 30621
2026-09-24T13:38:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2023 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T13:38:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:07Z Disabled control of system clock
I20260924 13:38:07.758693 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37447
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:37447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 13:38:07.838009 31151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:07.838207 31151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:07.838227 31151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:07.839331 31151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:07.839370 31151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:07.839383 31151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:07.839395 31151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:07.839406 31151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:07.840649 31151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:37447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:07.840847 31151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:07.841059 31151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:07.845731 31156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.845760 31159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.845762 31157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:07.845817 31151 server_base.cc:1034] running on GCE node
I20260924 13:38:07.846256 31151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:07.846547 31151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:07.847687 31151 hybrid_clock.cc:648] HybridClock initialized: now 1790257087847666 us; error 43 us; skew 500 ppm
I20260924 13:38:07.848845 31151 webserver.cc:460] Webserver started at http://127.25.240.126:38155/ using document root <none> and password file <none>
I20260924 13:38:07.849076 31151 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:07.849148 31151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:07.849304 31151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:07.850394 31151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "9cada5802ab749a486b6c2abbfebb675"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "ea790ffc7751448f475ffb27f64f8589"
server_key_iv: "29b50845e6b7f29fdd31275db2b6333b"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.850752 31151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "9cada5802ab749a486b6c2abbfebb675"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "ea790ffc7751448f475ffb27f64f8589"
server_key_iv: "29b50845e6b7f29fdd31275db2b6333b"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.852006 31151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:38:07.852710 31165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.852885 31151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:07.852988 31151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "9cada5802ab749a486b6c2abbfebb675"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "ea790ffc7751448f475ffb27f64f8589"
server_key_iv: "29b50845e6b7f29fdd31275db2b6333b"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.853062 31151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:07.856393 31151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:07.856652 31151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:07.856798 31151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:07.860679 31151 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:37447
I20260924 13:38:07.860710 31217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:37447 every 8 connection(s)
I20260924 13:38:07.861032 31151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:07.861699 31218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:07.862658 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 31151
I20260924 13:38:07.862757 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:07.862927 26561 external_mini_cluster.cc:1428] Setting key c05325d65d7b6ea56d75d10ddc65afa3
I20260924 13:38:07.865079 31218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675: Bootstrap starting.
I20260924 13:38:07.865370 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:07.863382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.25.240.126:37447)
0924 13:38:07.863596 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:07.863604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:07.863627 (+ 23us) server_negotiation.cc:407] Connection header received
0924 13:38:07.864027 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:07.864034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:07.864077 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:07.864227 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:07.864931 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:07.864936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:07.864937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:07.864945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:07.864982 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:07.864991 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:07.865065 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:07.865136 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 13:38:07.865196 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260924 13:38:07.865829 31218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:07.866112 31218 log.cc:824] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:07.866690 31218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675: No bootstrap required, opened a new log
I20260924 13:38:07.868036 31218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cada5802ab749a486b6c2abbfebb675" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37447 } }
I20260924 13:38:07.868162 31218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:07.868183 31218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cada5802ab749a486b6c2abbfebb675, State: Initialized, Role: FOLLOWER
I20260924 13:38:07.868261 31218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cada5802ab749a486b6c2abbfebb675" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37447 } }
I20260924 13:38:07.868317 31218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:07.868340 31218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:07.868384 31218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:07.869166 31218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cada5802ab749a486b6c2abbfebb675" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37447 } }
I20260924 13:38:07.869248 31218 leader_election.cc:302] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cada5802ab749a486b6c2abbfebb675; no voters:
I20260924 13:38:07.869361 31218 leader_election.cc:288] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:07.869434 31223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:07.869604 31223 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [term 1 LEADER]: Becoming Leader. State: Replica: 9cada5802ab749a486b6c2abbfebb675, State: Running, Role: LEADER
I20260924 13:38:07.869640 31218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:07.869705 31223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cada5802ab749a486b6c2abbfebb675" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37447 } }
I20260924 13:38:07.870114 31224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cada5802ab749a486b6c2abbfebb675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cada5802ab749a486b6c2abbfebb675" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37447 } } }
I20260924 13:38:07.870412 31224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:07.870330 31225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cada5802ab749a486b6c2abbfebb675. Latest consensus state: current_term: 1 leader_uuid: "9cada5802ab749a486b6c2abbfebb675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cada5802ab749a486b6c2abbfebb675" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37447 } } }
I20260924 13:38:07.870576 31225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:07.870723 31232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:07.871227 31232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:07.872875 31232 catalog_manager.cc:1347] Generated new cluster ID: 8ba894755d76445087142e65e0ad3208
I20260924 13:38:07.872923 31232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:07.879854 31232 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:07.880007 31232 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:07.883522 31232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675: Generated new TSK 0
I20260924 13:38:07.883678 31232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:38:07.889271 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37447
--builtin_ntp_servers=127.25.240.84:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 13:38:07.970135 31242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:07.970366 31242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:07.970400 31242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:07.971714 31242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:07.971778 31242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:07.971819 31242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:07.973178 31242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:07.973453 31242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:07.973824 31242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:07.978765 31250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:07.978777 31247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:07.978958 31242 server_base.cc:1034] running on GCE node
W20260924 13:38:07.979085 31248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:07.979315 31242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:07.979578 31242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:07.980726 31242 hybrid_clock.cc:648] HybridClock initialized: now 1790257087980708 us; error 35 us; skew 500 ppm
I20260924 13:38:07.982147 31242 webserver.cc:460] Webserver started at http://127.25.240.65:44073/ using document root <none> and password file <none>
I20260924 13:38:07.982417 31242 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:07.982463 31242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:07.982612 31242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:07.983664 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "563bea7ed9db49c9a20f4afb813eeb09"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "069a2b67c652758a710dad06de8fb19a"
server_key_iv: "b203e85e3cb0c967677f18edf2a9e81d"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.984100 31242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "563bea7ed9db49c9a20f4afb813eeb09"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "069a2b67c652758a710dad06de8fb19a"
server_key_iv: "b203e85e3cb0c967677f18edf2a9e81d"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.985416 31242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:07.986199 31256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.986378 31242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:07.986476 31242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "563bea7ed9db49c9a20f4afb813eeb09"
format_stamp: "Formatted at 2026-09-24 13:38:07 on dist-test-slave-m5z1"
server_key: "069a2b67c652758a710dad06de8fb19a"
server_key_iv: "b203e85e3cb0c967677f18edf2a9e81d"
server_key_version: "encryptionkey@0"
I20260924 13:38:07.986542 31242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:07.991508 31242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:07.991773 31242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:07.991894 31242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:07.992080 31242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:07.992377 31242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:07.992415 31242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.992442 31242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:07.992466 31242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:07.999330 31242 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:34867
I20260924 13:38:07.999408 31369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:34867 every 8 connection(s)
I20260924 13:38:07.999773 31242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:38:08.002731 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.000513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:50401 (local address 127.25.240.126:37447)
0924 13:38:08.000579 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.000582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.000789 (+ 207us) server_negotiation.cc:407] Connection header received
0924 13:38:08.001243 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.001245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.001251 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.001339 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.002546 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.002547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.002547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.002551 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.002564 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.002568 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.002619 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.002668 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.002680 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:38:08.002954 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.000484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37447 (local address 127.25.240.65:50401)
0924 13:38:08.000655 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.000695 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.001145 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.001399 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.001403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.001411 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:08.002402 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:08.002403 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:08.002433 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:08.002460 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:08.002620 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:08.002623 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:08.002625 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:38:08.002676 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:38:08.002721 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260924 13:38:08.003448 31370 heartbeater.cc:347] Connected to a master server at 127.25.240.126:37447
I20260924 13:38:08.003544 31370 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:08.003686 31370 heartbeater.cc:511] Master 127.25.240.126:37447 requested a full tablet report, sending...
I20260924 13:38:08.004177 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 31242
I20260924 13:38:08.004190 31182 ts_manager.cc:194] Registered new tserver with Master: 563bea7ed9db49c9a20f4afb813eeb09 (127.25.240.65:34867)
I20260924 13:38:08.004287 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:08.004460 26561 external_mini_cluster.cc:1428] Setting key 2cb0014dec785fa05b27872cf4a59bb0
I20260924 13:38:08.005334 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37447
--builtin_ntp_servers=127.25.240.84:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 13:38:08.085513 31373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:08.085738 31373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:08.085762 31373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:08.086944 31373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:08.086985 31373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:08.087033 31373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:08.088361 31373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:08.088610 31373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:08.088920 31373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:08.093924 31381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.093912 31379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.093940 31378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:08.094213 31373 server_base.cc:1034] running on GCE node
I20260924 13:38:08.094439 31373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:08.094630 31373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:08.095774 31373 hybrid_clock.cc:648] HybridClock initialized: now 1790257088095754 us; error 30 us; skew 500 ppm
I20260924 13:38:08.097024 31373 webserver.cc:460] Webserver started at http://127.25.240.66:42603/ using document root <none> and password file <none>
I20260924 13:38:08.097215 31373 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:08.097255 31373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:08.097347 31373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:08.098413 31373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "2c98282af3f1445383656f3dff144cd4"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "46a84aa5dc938533df59e5e25e5ee03b"
server_key_iv: "632648f0e31b15c8fba8729d38cdac91"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.098773 31373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "2c98282af3f1445383656f3dff144cd4"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "46a84aa5dc938533df59e5e25e5ee03b"
server_key_iv: "632648f0e31b15c8fba8729d38cdac91"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.099957 31373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:38:08.100607 31387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.100762 31373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:08.100835 31373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "2c98282af3f1445383656f3dff144cd4"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "46a84aa5dc938533df59e5e25e5ee03b"
server_key_iv: "632648f0e31b15c8fba8729d38cdac91"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.100883 31373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:08.104099 31373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:08.104319 31373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:08.104409 31373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:08.104561 31373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:08.104825 31373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:08.104861 31373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.104880 31373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:08.104893 31373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.110562 31373 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:45105
I20260924 13:38:08.110597 31500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:45105 every 8 connection(s)
I20260924 13:38:08.110927 31373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:38:08.113921 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.111645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:58819 (local address 127.25.240.126:37447)
0924 13:38:08.111688 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.111690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.111937 (+ 247us) server_negotiation.cc:407] Connection header received
0924 13:38:08.112434 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.112435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.112441 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.112500 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.113650 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.113652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.113652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.113657 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.113671 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.113675 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.113728 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.113835 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.113850 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:38:08.114108 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.111634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37447 (local address 127.25.240.66:58819)
0924 13:38:08.111814 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.111852 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.112327 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.112569 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.112573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.112597 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:08.113467 (+ 870us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:08.113467 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:08.113494 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:08.113521 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:08.113737 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:08.113742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:08.113745 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:08.113840 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 13:38:08.113903 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260924 13:38:08.114599 31501 heartbeater.cc:347] Connected to a master server at 127.25.240.126:37447
I20260924 13:38:08.114674 31501 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:08.114845 31501 heartbeater.cc:511] Master 127.25.240.126:37447 requested a full tablet report, sending...
I20260924 13:38:08.115206 31182 ts_manager.cc:194] Registered new tserver with Master: 2c98282af3f1445383656f3dff144cd4 (127.25.240.66:45105)
I20260924 13:38:08.119831 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 31373
I20260924 13:38:08.119939 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:08.120122 26561 external_mini_cluster.cc:1428] Setting key 6c82608ff6b9af19f573cfc87474ca11
I20260924 13:38:08.121011 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37447
--builtin_ntp_servers=127.25.240.84:44753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 13:38:08.204830 31504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:08.205061 31504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:08.205096 31504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:08.206260 31504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:08.206326 31504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:08.206367 31504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:08.207680 31504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:37447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31504
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:08.207935 31504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:08.208220 31504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:08.213119 31510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.213116 31512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.213112 31509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:08.213785 31504 server_base.cc:1034] running on GCE node
I20260924 13:38:08.213997 31504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:08.214187 31504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:08.215342 31504 hybrid_clock.cc:648] HybridClock initialized: now 1790257088215324 us; error 26 us; skew 500 ppm
I20260924 13:38:08.216634 31504 webserver.cc:460] Webserver started at http://127.25.240.67:35869/ using document root <none> and password file <none>
I20260924 13:38:08.216876 31504 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:08.216948 31504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:08.217101 31504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:08.218225 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "a618e3d69990487b95ae40fc9923b650"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "2343118179f66b9cdd1a0d838b6fbb50"
server_key_iv: "9255b62b23abe91ab0837e16cf24eee6"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.218592 31504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "a618e3d69990487b95ae40fc9923b650"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "2343118179f66b9cdd1a0d838b6fbb50"
server_key_iv: "9255b62b23abe91ab0837e16cf24eee6"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.219815 31504 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 13:38:08.220476 31518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.220629 31504 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.220729 31504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "a618e3d69990487b95ae40fc9923b650"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "2343118179f66b9cdd1a0d838b6fbb50"
server_key_iv: "9255b62b23abe91ab0837e16cf24eee6"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.220803 31504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:08.224133 31504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:08.224385 31504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:08.224525 31504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:08.224733 31504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:08.225080 31504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:08.225121 31504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.225160 31504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:08.225203 31504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.230712 31504 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:35989
I20260924 13:38:08.230770 31631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:35989 every 8 connection(s)
I20260924 13:38:08.231074 31504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:38:08.234066 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.231735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:36811 (local address 127.25.240.126:37447)
0924 13:38:08.231780 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.231783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.231992 (+ 209us) server_negotiation.cc:407] Connection header received
0924 13:38:08.232495 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.232497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.232504 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.232555 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.233825 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.233827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.233827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.233831 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.233846 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.233850 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.233908 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.233984 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.233998 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:38:08.234256 31634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.231714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37447 (local address 127.25.240.67:36811)
0924 13:38:08.231873 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.231905 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.232373 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.232632 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.232636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.232646 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:08.233660 (+ 1014us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:08.233660 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:08.233688 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:08.233714 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:08.233909 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:08.233914 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:08.233917 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:08.233994 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 13:38:08.234053 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260924 13:38:08.234733 31632 heartbeater.cc:347] Connected to a master server at 127.25.240.126:37447
I20260924 13:38:08.234822 31632 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:08.234926 31632 heartbeater.cc:511] Master 127.25.240.126:37447 requested a full tablet report, sending...
I20260924 13:38:08.235251 31182 ts_manager.cc:194] Registered new tserver with Master: a618e3d69990487b95ae40fc9923b650 (127.25.240.67:35989)
I20260924 13:38:08.235459 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 31504
I20260924 13:38:08.235534 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:08.235720 26561 external_mini_cluster.cc:1428] Setting key 09693bab53dc41b6f73027a9a145917a
I20260924 13:38:08.236502 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:38:08.238780 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.237840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.25.240.126:37447)
0924 13:38:08.237884 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.237886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.238000 (+ 114us) server_negotiation.cc:407] Connection header received
0924 13:38:08.238050 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.238051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.238054 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.238115 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.238552 (+ 437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.238552 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.238553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.238556 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.238567 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.238570 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.238623 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.238653 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.238664 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:38:08.240361 31182 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:38:08.242168 31182 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:08.246536 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.244360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:35989 (local address 127.0.0.1:45544)
0924 13:38:08.245366 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.245375 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.245418 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.245574 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.245575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.245578 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:08.246131 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:08.246131 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:08.246145 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:08.246162 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:08.246379 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:08.246384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:08.246388 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:08.246452 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:38:08.246467 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"thread_start_us":101,"threads_started":1}
I20260924 13:38:08.246547 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.244582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.25.240.67:35989)
0924 13:38:08.244712 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.244716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.245436 (+ 720us) server_negotiation.cc:407] Connection header received
0924 13:38:08.245464 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.245466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.245477 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.245551 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.246242 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.246248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.246248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.246252 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.246283 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.246301 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.246385 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.246455 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.246470 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260924 13:38:08.246992 31640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.244346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:45105 (local address 127.0.0.1:59658)
0924 13:38:08.245233 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.245244 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.245291 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.245902 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.245903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.245906 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:08.246297 (+ 391us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:08.246298 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:08.246306 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:08.246320 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:08.246555 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:08.246556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:08.246557 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:38:08.246905 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 13:38:08.246918 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":75,"threads_started":1}
I20260924 13:38:08.247198 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.244507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.25.240.65:34867)
0924 13:38:08.244660 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.244666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.245094 (+ 428us) server_negotiation.cc:407] Connection header received
0924 13:38:08.245213 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.245219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.245231 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.245309 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.246840 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.246842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.246843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.246846 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.246877 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.246891 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.246955 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.247029 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.247046 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260924 13:38:08.247260 31566 tablet_service.cc:1467] Processing CreateTablet for tablet f908520f23fe48beb34d2b6b1b62186f (DEFAULT_TABLE table=test-table [id=6cfbda5a7d544f029b782dc5ed5100b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:08.247543 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.244693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.25.240.66:45105)
0924 13:38:08.244847 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.244850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.245472 (+ 622us) server_negotiation.cc:407] Connection header received
0924 13:38:08.245502 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.245504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.245514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.245609 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.246406 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.246414 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.246414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.246419 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.246453 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.246475 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.246530 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.247371 (+ 841us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.247417 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260924 13:38:08.247604 31566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f908520f23fe48beb34d2b6b1b62186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:08.247093 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.244339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:34867 (local address 127.0.0.1:33562)
0924 13:38:08.244851 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.244882 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.245107 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.245520 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.245526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.245533 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:08.246440 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:08.246441 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:08.246447 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:08.246456 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:08.246961 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:08.246962 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:08.246963 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:38:08.247022 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 13:38:08.247035 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":73,"threads_started":1}
I20260924 13:38:08.248332 31304 tablet_service.cc:1467] Processing CreateTablet for tablet f908520f23fe48beb34d2b6b1b62186f (DEFAULT_TABLE table=test-table [id=6cfbda5a7d544f029b782dc5ed5100b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:08.248499 31435 tablet_service.cc:1467] Processing CreateTablet for tablet f908520f23fe48beb34d2b6b1b62186f (DEFAULT_TABLE table=test-table [id=6cfbda5a7d544f029b782dc5ed5100b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:08.248631 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f908520f23fe48beb34d2b6b1b62186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:08.248835 31435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f908520f23fe48beb34d2b6b1b62186f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:08.250883 31646 tablet_bootstrap.cc:492] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: Bootstrap starting.
I20260924 13:38:08.251582 31647 tablet_bootstrap.cc:492] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09: Bootstrap starting.
I20260924 13:38:08.251641 31646 tablet_bootstrap.cc:654] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:08.252080 31646 log.cc:824] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:08.252174 31648 tablet_bootstrap.cc:492] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4: Bootstrap starting.
I20260924 13:38:08.252251 31647 tablet_bootstrap.cc:654] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:08.252561 31647 log.cc:824] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:08.252851 31648 tablet_bootstrap.cc:654] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:08.253067 31646 tablet_bootstrap.cc:492] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: No bootstrap required, opened a new log
I20260924 13:38:08.253139 31646 ts_tablet_manager.cc:1397] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:08.253185 31648 log.cc:824] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:08.253253 31647 tablet_bootstrap.cc:492] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09: No bootstrap required, opened a new log
I20260924 13:38:08.253317 31647 ts_tablet_manager.cc:1397] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:08.253862 31648 tablet_bootstrap.cc:492] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4: No bootstrap required, opened a new log
I20260924 13:38:08.253947 31648 ts_tablet_manager.cc:1397] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:08.255800 31648 raft_consensus.cc:348] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.255975 31648 raft_consensus.cc:374] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:08.255870 31647 raft_consensus.cc:348] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.255862 31646 raft_consensus.cc:348] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.256019 31648 raft_consensus.cc:729] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c98282af3f1445383656f3dff144cd4, State: Initialized, Role: FOLLOWER
I20260924 13:38:08.256021 31647 raft_consensus.cc:374] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:08.256032 31646 raft_consensus.cc:374] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:08.256065 31646 raft_consensus.cc:729] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a618e3d69990487b95ae40fc9923b650, State: Initialized, Role: FOLLOWER
I20260924 13:38:08.256062 31647 raft_consensus.cc:729] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 563bea7ed9db49c9a20f4afb813eeb09, State: Initialized, Role: FOLLOWER
I20260924 13:38:08.256134 31648 consensus_queue.cc:260] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.256213 31647 consensus_queue.cc:260] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.256413 31648 ts_tablet_manager.cc:1428] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 13:38:08.256482 31647 ts_tablet_manager.cc:1428] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:38:08.256441 31646 consensus_queue.cc:260] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.256538 31501 heartbeater.cc:503] Master 127.25.240.126:37447 was elected leader, sending a full tablet report...
I20260924 13:38:08.256592 31370 heartbeater.cc:503] Master 127.25.240.126:37447 was elected leader, sending a full tablet report...
I20260924 13:38:08.256687 31646 ts_tablet_manager.cc:1428] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 13:38:08.256706 31632 heartbeater.cc:503] Master 127.25.240.126:37447 was elected leader, sending a full tablet report...
I20260924 13:38:08.358657 31652 raft_consensus.cc:482] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:08.358814 31652 raft_consensus.cc:504] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.359215 31652 leader_election.cc:288] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 563bea7ed9db49c9a20f4afb813eeb09 (127.25.240.65:34867), a618e3d69990487b95ae40fc9923b650 (127.25.240.67:35989)
I20260924 13:38:08.360780 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.359445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:34867 (local address 127.25.240.66:38245)
0924 13:38:08.359498 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.359510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.359555 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.359742 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.359743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.359746 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:08.360367 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:08.360368 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:08.360376 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:08.360389 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:08.360609 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:08.360611 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:08.360612 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:38:08.360674 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 13:38:08.360690 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:38:08.360807 31643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.359563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:38245 (local address 127.25.240.65:34867)
0924 13:38:08.359609 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.359611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.359619 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:08.359655 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.359656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.359661 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.359713 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.360481 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.360482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.360483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.360488 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.360508 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.360513 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.360569 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.360709 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.360726 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:38:08.360855 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.359527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:42139 (local address 127.25.240.67:35989)
0924 13:38:08.359582 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.359585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.359761 (+ 176us) server_negotiation.cc:407] Connection header received
0924 13:38:08.359956 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.359958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.359966 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.360049 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.360624 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.360626 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.360626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.360631 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.360648 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.360653 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.360709 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.360767 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.360779 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 13:38:08.360904 31655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.359446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:35989 (local address 127.25.240.66:42139)
0924 13:38:08.359667 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.359675 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.359744 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.360066 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.360067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.360070 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:08.360533 (+ 463us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:08.360533 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:08.360539 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:08.360550 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:08.360718 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:08.360719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:08.360720 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:38:08.360769 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:38:08.360781 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":61,"thread_start_us":109,"threads_started":1}
I20260924 13:38:08.361245 31586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f908520f23fe48beb34d2b6b1b62186f" candidate_uuid: "2c98282af3f1445383656f3dff144cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a618e3d69990487b95ae40fc9923b650" is_pre_election: true
I20260924 13:38:08.361320 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f908520f23fe48beb34d2b6b1b62186f" candidate_uuid: "2c98282af3f1445383656f3dff144cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563bea7ed9db49c9a20f4afb813eeb09" is_pre_election: true
I20260924 13:38:08.361405 31586 raft_consensus.cc:2454] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c98282af3f1445383656f3dff144cd4 in term 0.
I20260924 13:38:08.361433 31324 raft_consensus.cc:2454] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c98282af3f1445383656f3dff144cd4 in term 0.
W20260924 13:38:08.361469 31502 tablet.cc:2369] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:38:08.361670 31389 leader_election.cc:302] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c98282af3f1445383656f3dff144cd4, a618e3d69990487b95ae40fc9923b650; no voters:
I20260924 13:38:08.361784 31652 raft_consensus.cc:2781] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:08.361846 31652 raft_consensus.cc:482] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:08.361871 31652 raft_consensus.cc:3037] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:08.362565 31652 raft_consensus.cc:504] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.362711 31652 leader_election.cc:288] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [CANDIDATE]: Term 1 election: Requested vote from peers 563bea7ed9db49c9a20f4afb813eeb09 (127.25.240.65:34867), a618e3d69990487b95ae40fc9923b650 (127.25.240.67:35989)
I20260924 13:38:08.362883 31324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f908520f23fe48beb34d2b6b1b62186f" candidate_uuid: "2c98282af3f1445383656f3dff144cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "563bea7ed9db49c9a20f4afb813eeb09"
I20260924 13:38:08.362893 31586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f908520f23fe48beb34d2b6b1b62186f" candidate_uuid: "2c98282af3f1445383656f3dff144cd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a618e3d69990487b95ae40fc9923b650"
I20260924 13:38:08.362972 31586 raft_consensus.cc:3037] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:08.362985 31324 raft_consensus.cc:3037] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:08.363670 31324 raft_consensus.cc:2454] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c98282af3f1445383656f3dff144cd4 in term 1.
I20260924 13:38:08.363705 31586 raft_consensus.cc:2454] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c98282af3f1445383656f3dff144cd4 in term 1.
I20260924 13:38:08.363878 31388 leader_election.cc:302] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c98282af3f1445383656f3dff144cd4, 563bea7ed9db49c9a20f4afb813eeb09; no voters:
I20260924 13:38:08.363984 31652 raft_consensus.cc:2781] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:08.364147 31652 raft_consensus.cc:686] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 1 LEADER]: Becoming Leader. State: Replica: 2c98282af3f1445383656f3dff144cd4, State: Running, Role: LEADER
I20260924 13:38:08.364251 31652 consensus_queue.cc:237] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } }
I20260924 13:38:08.365031 31180 catalog_manager.cc:5495] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c98282af3f1445383656f3dff144cd4 (127.25.240.66). New cstate: current_term: 1 leader_uuid: "2c98282af3f1445383656f3dff144cd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c98282af3f1445383656f3dff144cd4" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 } health_report { overall_health: UNKNOWN } } }
I20260924 13:38:08.392467 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.391623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59662 (local address 127.25.240.66:45105)
0924 13:38:08.391701 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.391703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.391710 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:08.391746 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.391747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.391760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.391815 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.392303 (+ 488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.392304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.392304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.392307 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.392321 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.392325 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.392369 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.392400 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.392412 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:38:08.394230 31324 raft_consensus.cc:1261] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 1 FOLLOWER]: Refusing update from remote peer 2c98282af3f1445383656f3dff144cd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:08.394246 31586 raft_consensus.cc:1261] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 1 FOLLOWER]: Refusing update from remote peer 2c98282af3f1445383656f3dff144cd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:08.394609 31652 consensus_queue.cc:1035] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "563bea7ed9db49c9a20f4afb813eeb09" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:08.394788 31656 consensus_queue.cc:1035] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a618e3d69990487b95ae40fc9923b650" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:08.397811 31660 mvcc.cc:205] Tried to move back new op lower bound from 7332893034060111872 to 7332893033940643840. Current Snapshot: MvccSnapshot[applied={T|T < 7332893034060111872}]
I20260924 13:38:08.399631 31180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:08.399734 31180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:08.400506 31180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cada5802ab749a486b6c2abbfebb675: Sending DeleteTablet for 3 replicas of tablet f908520f23fe48beb34d2b6b1b62186f
I20260924 13:38:08.400817 31435 tablet_service.cc:1514] Processing DeleteTablet for tablet f908520f23fe48beb34d2b6b1b62186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:08 UTC) from {username='slave'} at 127.0.0.1:59658
I20260924 13:38:08.400918 31304 tablet_service.cc:1514] Processing DeleteTablet for tablet f908520f23fe48beb34d2b6b1b62186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:08 UTC) from {username='slave'} at 127.0.0.1:33562
I20260924 13:38:08.401026 31566 tablet_service.cc:1514] Processing DeleteTablet for tablet f908520f23fe48beb34d2b6b1b62186f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:08 UTC) from {username='slave'} at 127.0.0.1:45544
I20260924 13:38:08.401168 31669 tablet_replica.cc:330] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09: stopping tablet replica
I20260924 13:38:08.401270 31669 raft_consensus.cc:2229] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:08.401342 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 31242
I20260924 13:38:08.401383 31669 raft_consensus.cc:2258] T f908520f23fe48beb34d2b6b1b62186f P 563bea7ed9db49c9a20f4afb813eeb09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:08.406919 31670 tablet_replica.cc:330] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: stopping tablet replica
W20260924 13:38:08.406932 31166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:34867 (error 108)
I20260924 13:38:08.407047 31670 raft_consensus.cc:2229] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:08.407166 31670 raft_consensus.cc:2258] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:08.407298 31668 tablet_replica.cc:330] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4: stopping tablet replica
I20260924 13:38:08.407441 31668 raft_consensus.cc:2229] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:38:08.407580 31668 raft_consensus.cc:2258] T f908520f23fe48beb34d2b6b1b62186f P 2c98282af3f1445383656f3dff144cd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:08.407604 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.407463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:34867 (local address 127.0.0.1:33564)
0924 13:38:08.407524 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.407569 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:38:08.407593 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 31373
W20260924 13:38:08.408968 31166 catalog_manager.cc:4584] TS 563bea7ed9db49c9a20f4afb813eeb09 (127.25.240.65:34867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f908520f23fe48beb34d2b6b1b62186f: Network error: Client connection negotiation failed: client connection to 127.25.240.65:34867: connect: Connection refused (error 111)
I20260924 13:38:08.409313 31670 ts_tablet_manager.cc:1905] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:08.411016 31670 ts_tablet_manager.cc:1918] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:08.411103 31670 log.cc:1196] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/f908520f23fe48beb34d2b6b1b62186f
I20260924 13:38:08.411396 31670 ts_tablet_manager.cc:1939] T f908520f23fe48beb34d2b6b1b62186f P a618e3d69990487b95ae40fc9923b650: Deleting consensus metadata
I20260924 13:38:08.411816 31167 catalog_manager.cc:4841] TS a618e3d69990487b95ae40fc9923b650 (127.25.240.67:35989): tablet f908520f23fe48beb34d2b6b1b62186f (table test-table [id=6cfbda5a7d544f029b782dc5ed5100b1]) successfully deleted
I20260924 13:38:08.412882 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 31504
I20260924 13:38:08.413014 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.412865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:45105 (local address 127.0.0.1:59678)
0924 13:38:08.412931 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.412960 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.66:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 13:38:08.417399 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 31151
2026-09-24T13:38:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (685 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 13:38:08.436954 26561 mini_oidc.cc:152] Starting JWKS server
I20260924 13:38:08.438126 26561 webserver.cc:460] Webserver started at https://127.0.0.1:35921/ using document root <none> and password file <none>
I20260924 13:38:08.438249 26561 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 13:38:08.438441 26561 webserver.cc:460] Webserver started at http://127.0.0.1:41185/ using document root <none> and password file <none>
2026-09-24T13:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:08Z Disabled control of system clock
I20260924 13:38:08.446127 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:41819
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:41819
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35921/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/testchainca.cert with env {}
W20260924 13:38:08.524811 31677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:08.525003 31677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:08.525027 31677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:08.526149 31677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:08.526183 31677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:08.526196 31677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:08.526208 31677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:08.526221 31677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:38:08.526232 31677 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:38:08.526244 31677 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35921/jwks.json
I20260924 13:38:08.527719 31677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:41819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35921/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:08.527910 31677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:08.528066 31677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:38:08.532887 31677 server_base.cc:1034] running on GCE node
W20260924 13:38:08.532857 31682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.532857 31683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.532882 31685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:08.533344 31677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:08.533687 31677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:08.534875 31677 hybrid_clock.cc:648] HybridClock initialized: now 1790257088534858 us; error 45 us; skew 500 ppm
I20260924 13:38:08.536226 31677 webserver.cc:460] Webserver started at http://127.25.240.126:38289/ using document root <none> and password file <none>
I20260924 13:38:08.536484 31677 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:08.536556 31677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:08.536695 31677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:08.537815 31677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "365fff7e1b85424eb7d24286bc5eb16f"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "1e130d926c89b596d1d72039933436d8"
server_key_iv: "0e01181fa661e2e0fb424e772b17d055"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.538174 31677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "365fff7e1b85424eb7d24286bc5eb16f"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "1e130d926c89b596d1d72039933436d8"
server_key_iv: "0e01181fa661e2e0fb424e772b17d055"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.539446 31677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:08.540212 31691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.540390 31677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:08.540488 31677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "365fff7e1b85424eb7d24286bc5eb16f"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "1e130d926c89b596d1d72039933436d8"
server_key_iv: "0e01181fa661e2e0fb424e772b17d055"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.540570 31677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:08.551039 31677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:08.551343 31677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:08.551494 31677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:08.555361 31677 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:41819
I20260924 13:38:08.555392 31743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:41819 every 8 connection(s)
I20260924 13:38:08.555727 31677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:08.556676 31744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:08.558981 31744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f: Bootstrap starting.
I20260924 13:38:08.559559 31744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:08.559852 31744 log.cc:824] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:08.560365 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 31677
I20260924 13:38:08.560474 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:08.560608 31744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f: No bootstrap required, opened a new log
I20260924 13:38:08.560654 26561 external_mini_cluster.cc:1428] Setting key 343927b846a39fbcfbfd0a13b91e1cf2
I20260924 13:38:08.563419 31744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365fff7e1b85424eb7d24286bc5eb16f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41819 } }
I20260924 13:38:08.563583 31744 raft_consensus.cc:374] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:08.563612 31744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 365fff7e1b85424eb7d24286bc5eb16f, State: Initialized, Role: FOLLOWER
I20260924 13:38:08.563723 31744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365fff7e1b85424eb7d24286bc5eb16f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41819 } }
I20260924 13:38:08.563793 31744 raft_consensus.cc:388] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:08.563830 31744 raft_consensus.cc:482] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:08.563890 31744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:08.564463 31744 raft_consensus.cc:504] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365fff7e1b85424eb7d24286bc5eb16f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41819 } }
I20260924 13:38:08.564581 31744 leader_election.cc:302] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 365fff7e1b85424eb7d24286bc5eb16f; no voters:
I20260924 13:38:08.564764 31744 leader_election.cc:288] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:08.564826 31749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:08.564997 31749 raft_consensus.cc:686] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [term 1 LEADER]: Becoming Leader. State: Replica: 365fff7e1b85424eb7d24286bc5eb16f, State: Running, Role: LEADER
I20260924 13:38:08.565019 31744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:08.565088 31749 consensus_queue.cc:237] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365fff7e1b85424eb7d24286bc5eb16f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41819 } }
I20260924 13:38:08.565392 31749 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 365fff7e1b85424eb7d24286bc5eb16f. Latest consensus state: current_term: 1 leader_uuid: "365fff7e1b85424eb7d24286bc5eb16f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365fff7e1b85424eb7d24286bc5eb16f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41819 } } }
I20260924 13:38:08.565485 31749 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:08.565403 31751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "365fff7e1b85424eb7d24286bc5eb16f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "365fff7e1b85424eb7d24286bc5eb16f" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41819 } } }
I20260924 13:38:08.565598 31751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:08.565718 31756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:08.566198 31756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:08.567148 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.561150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.25.240.126:41819)
0924 13:38:08.561385 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.561395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.561427 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:38:08.561915 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.561924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.562000 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.562174 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.562370 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.563765 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.564659 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.565284 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.565819 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:08.566214 (+ 395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.566219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.566219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.566562 (+ 343us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.566615 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.566632 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.566908 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.566935 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.566997 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":104,"threads_started":1}
I20260924 13:38:08.568084 31756 catalog_manager.cc:1347] Generated new cluster ID: 3f8c9f9a40494975b7a7daf08f72520e
I20260924 13:38:08.568135 31756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:08.568120 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:38:08.576661 31756 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:08.577332 31756 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:08.581362 31756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 365fff7e1b85424eb7d24286bc5eb16f: Generated new TSK 0
I20260924 13:38:08.581521 31756 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 13:38:08.611663 31774 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 13:38:08.611757 31774 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 13:38:08.611923 31748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.588726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57372 (local address 127.25.240.126:41819)
0924 13:38:08.588788 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.588791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.588938 (+ 147us) server_negotiation.cc:407] Connection header received
0924 13:38:08.589061 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.589062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.589074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.589139 (+ 65us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:38:08.589294 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.590104 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.590916 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.591173 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.591255 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:08.591480 (+ 225us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:38:08.611644 (+ 20164us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 13:38:08.611848 (+ 204us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57372: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 13:38:08.611994 31748 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57372: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 13:38:08.612366 31773 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:08.588665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:41819 (local address 127.0.0.1:57372)
0924 13:38:08.588870 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.588879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.588959 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.589145 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.589146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.589148 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:38:08.589211 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:08.589216 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.590205 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.590206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:08.590810 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:08.590812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.591265 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.591267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:08.591345 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:08.591365 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:38:08.611945 (+ 20580us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 13:38:08.612083 (+ 138us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.240.126:41819: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.25.240.126:41819: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35921/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 13:38:08.612802 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 31677
2026-09-24T13:38:08Z chronyd exiting
I20260924 13:38:08.648315 26561 test_util.cc:175] -----------------------------------------------
I20260924 13:38:08.648416 26561 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790257062855231-26561-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (212 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 13:38:08.649133 26561 mini_oidc.cc:152] Starting JWKS server
I20260924 13:38:08.649887 26561 webserver.cc:460] Webserver started at https://127.0.0.1:45173/ using document root <none> and password file <none>
I20260924 13:38:08.649957 26561 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 13:38:08.650123 26561 webserver.cc:460] Webserver started at http://127.0.0.1:35273/ using document root <none> and password file <none>
2026-09-24T13:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:08Z Disabled control of system clock
I20260924 13:38:08.657859 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:45393
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:41425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:45393
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45173/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/testchainca.cert with env {}
W20260924 13:38:08.742583 31780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:08.742798 31780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:08.742820 31780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:08.743983 31780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:08.744022 31780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:08.744035 31780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:08.744047 31780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:08.744059 31780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:38:08.744072 31780 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:38:08.744083 31780 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45173/jwks.json
I20260924 13:38:08.745533 31780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:45393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45173/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:08.745735 31780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:08.745893 31780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:08.750512 31785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.750512 31786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.750543 31788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:08.750746 31780 server_base.cc:1034] running on GCE node
I20260924 13:38:08.751062 31780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:08.751360 31780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:08.752526 31780 hybrid_clock.cc:648] HybridClock initialized: now 1790257088752523 us; error 38 us; skew 500 ppm
I20260924 13:38:08.753769 31780 webserver.cc:460] Webserver started at http://127.25.240.126:42925/ using document root <none> and password file <none>
I20260924 13:38:08.754009 31780 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:08.754076 31780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:08.754226 31780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:08.755246 31780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "e9dce24a0492402c9642a0216b830e9b"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "99cc1c516d5dc2d762538c3020a16e26"
server_key_iv: "6cf47b02a25f9b3fa2b092dafb6c0934"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.755617 31780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "e9dce24a0492402c9642a0216b830e9b"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "99cc1c516d5dc2d762538c3020a16e26"
server_key_iv: "6cf47b02a25f9b3fa2b092dafb6c0934"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.757059 31780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:08.757923 31794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.758100 31780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:08.758205 31780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "e9dce24a0492402c9642a0216b830e9b"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "99cc1c516d5dc2d762538c3020a16e26"
server_key_iv: "6cf47b02a25f9b3fa2b092dafb6c0934"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.758294 31780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:08.771636 31780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:08.771967 31780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:08.772137 31780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:08.776188 31780 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:45393
I20260924 13:38:08.776221 31846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:45393 every 8 connection(s)
I20260924 13:38:08.776588 31780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:08.777199 31847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:08.779841 31847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b: Bootstrap starting.
I20260924 13:38:08.780352 31847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:08.780578 31847 log.cc:824] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:08.781175 31847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b: No bootstrap required, opened a new log
I20260924 13:38:08.782744 31847 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dce24a0492402c9642a0216b830e9b" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45393 } }
I20260924 13:38:08.782888 31847 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:08.782912 31847 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9dce24a0492402c9642a0216b830e9b, State: Initialized, Role: FOLLOWER
I20260924 13:38:08.783010 31847 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dce24a0492402c9642a0216b830e9b" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45393 } }
I20260924 13:38:08.783061 31847 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:08.783082 31847 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:08.783106 31847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:08.783566 31847 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dce24a0492402c9642a0216b830e9b" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45393 } }
I20260924 13:38:08.783648 31847 leader_election.cc:302] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9dce24a0492402c9642a0216b830e9b; no voters:
I20260924 13:38:08.783840 31847 leader_election.cc:288] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:08.783898 31850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:08.784070 31850 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [term 1 LEADER]: Becoming Leader. State: Replica: e9dce24a0492402c9642a0216b830e9b, State: Running, Role: LEADER
I20260924 13:38:08.784101 31847 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:08.784165 31850 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dce24a0492402c9642a0216b830e9b" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45393 } }
I20260924 13:38:08.784391 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 31780
I20260924 13:38:08.784482 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:08.784699 26561 external_mini_cluster.cc:1428] Setting key b3e6367b4777e8fd4879a61a0a8b440c
I20260924 13:38:08.785141 31851 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9dce24a0492402c9642a0216b830e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dce24a0492402c9642a0216b830e9b" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45393 } } }
I20260924 13:38:08.785235 31851 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:08.785410 31852 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9dce24a0492402c9642a0216b830e9b. Latest consensus state: current_term: 1 leader_uuid: "e9dce24a0492402c9642a0216b830e9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9dce24a0492402c9642a0216b830e9b" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45393 } } }
I20260924 13:38:08.785475 31852 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:08.785802 31864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:08.786301 31864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:08.788663 31864 catalog_manager.cc:1347] Generated new cluster ID: 01f845a9ec6043e0b352041974974f44
I20260924 13:38:08.788723 31864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:08.790584 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.785500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.25.240.126:45393)
0924 13:38:08.785730 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.785739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.785772 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:38:08.785905 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.785913 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.785983 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.786124 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:08.786306 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.787503 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.788601 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.789067 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.789205 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:08.789920 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:08.789924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:08.789924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:08.790100 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:08.790133 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:08.790144 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:08.790217 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:08.790296 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 13:38:08.790388 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260924 13:38:08.791501 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:38:08.811034 31864 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:08.811664 31864 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:08.822885 31864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9dce24a0492402c9642a0216b830e9b: Generated new TSK 0
I20260924 13:38:08.823096 31864 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 13:38:08.850740 31877 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 13:38:08.850826 31877 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 13:38:08.851150 31868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:08.837052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.25.240.126:45393)
0924 13:38:08.837121 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:08.837123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:08.837135 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:38:08.837207 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:08.837207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:08.837220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:08.837319 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:38:08.837444 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.838242 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.839018 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.839346 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.839416 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:08.839541 (+ 125us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:38:08.850928 (+ 11387us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 13:38:08.851080 (+ 152us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 13:38:08.851215 31868 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 13:38:08.851192 31876 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:08.836923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:45393 (local address 127.0.0.1:57142)
0924 13:38:08.837082 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:08.837093 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:08.837145 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:08.837317 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:08.837317 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:08.837319 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:38:08.837377 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:08.837379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.838350 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.838351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:08.838932 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:08.838933 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:08.839421 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:08.839422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:08.839462 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:08.839466 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:38:08.851029 (+ 11563us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 13:38:08.851121 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.240.126:45393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45173/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260924 13:38:08.851486 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 31780
2026-09-24T13:38:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (239 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 13:38:08.888907 26561 mini_oidc.cc:152] Starting JWKS server
I20260924 13:38:08.889657 26561 webserver.cc:460] Webserver started at https://127.0.0.1:39253/ using document root <none> and password file <none>
I20260924 13:38:08.889726 26561 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 13:38:08.889918 26561 webserver.cc:460] Webserver started at http://127.0.0.1:44887/ using document root <none> and password file <none>
2026-09-24T13:38:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:08Z Disabled control of system clock
I20260924 13:38:08.897516 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38987
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:37243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:38987
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39253/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 13:38:08.976219 31883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:08.976428 31883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:08.976450 31883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:08.977684 31883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:08.977732 31883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:08.977746 31883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:08.977758 31883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:08.977770 31883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:38:08.977782 31883 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:38:08.977794 31883 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39253/jwks.json
I20260924 13:38:08.979142 31883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:38987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39253/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:08.979367 31883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:08.979632 31883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:08.984313 31889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:08.984367 31888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:08.984470 31883 server_base.cc:1034] running on GCE node
W20260924 13:38:08.984340 31891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:08.984758 31883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:08.985041 31883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:08.986181 31883 hybrid_clock.cc:648] HybridClock initialized: now 1790257088986160 us; error 43 us; skew 500 ppm
I20260924 13:38:08.987329 31883 webserver.cc:460] Webserver started at http://127.25.240.126:46079/ using document root <none> and password file <none>
I20260924 13:38:08.987520 31883 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:08.987561 31883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:08.987649 31883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:08.988631 31883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "5e07a3fc96dc45268dce6622d2ae9f93"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "8aa1dbd36e293aaf4b04b1a9bd4cba08"
server_key_iv: "c36dab4e4ba748d0bbc3c157017ed787"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.988922 31883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "5e07a3fc96dc45268dce6622d2ae9f93"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "8aa1dbd36e293aaf4b04b1a9bd4cba08"
server_key_iv: "c36dab4e4ba748d0bbc3c157017ed787"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.990257 31883 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:08.990913 31897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:08.991086 31883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:08.991154 31883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "5e07a3fc96dc45268dce6622d2ae9f93"
format_stamp: "Formatted at 2026-09-24 13:38:08 on dist-test-slave-m5z1"
server_key: "8aa1dbd36e293aaf4b04b1a9bd4cba08"
server_key_iv: "c36dab4e4ba748d0bbc3c157017ed787"
server_key_version: "encryptionkey@0"
I20260924 13:38:08.991204 31883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:09.002164 31883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:09.002401 31883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:09.002498 31883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:09.006412 31883 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:38987
I20260924 13:38:09.006443 31949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:38987 every 8 connection(s)
I20260924 13:38:09.006768 31883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:09.007383 31950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:09.010244 31950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93: Bootstrap starting.
I20260924 13:38:09.010823 31950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:09.011109 31950 log.cc:824] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:09.011765 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 31883
I20260924 13:38:09.011785 31950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93: No bootstrap required, opened a new log
I20260924 13:38:09.011852 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:09.012074 26561 external_mini_cluster.cc:1428] Setting key a08bf1f944031085612e9b8397669022
I20260924 13:38:09.014035 31950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07a3fc96dc45268dce6622d2ae9f93" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38987 } }
I20260924 13:38:09.014207 31950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:09.014245 31950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e07a3fc96dc45268dce6622d2ae9f93, State: Initialized, Role: FOLLOWER
I20260924 13:38:09.014379 31950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07a3fc96dc45268dce6622d2ae9f93" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38987 } }
I20260924 13:38:09.014495 31950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:09.014528 31950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:09.014575 31950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:09.015297 31950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07a3fc96dc45268dce6622d2ae9f93" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38987 } }
I20260924 13:38:09.015411 31950 leader_election.cc:302] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e07a3fc96dc45268dce6622d2ae9f93; no voters:
I20260924 13:38:09.015555 31950 leader_election.cc:288] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:09.015599 31955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:09.015750 31955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [term 1 LEADER]: Becoming Leader. State: Replica: 5e07a3fc96dc45268dce6622d2ae9f93, State: Running, Role: LEADER
I20260924 13:38:09.015784 31950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:09.015846 31955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07a3fc96dc45268dce6622d2ae9f93" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38987 } }
I20260924 13:38:09.016157 31956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e07a3fc96dc45268dce6622d2ae9f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07a3fc96dc45268dce6622d2ae9f93" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38987 } } }
I20260924 13:38:09.016244 31956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:09.016404 31957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e07a3fc96dc45268dce6622d2ae9f93. Latest consensus state: current_term: 1 leader_uuid: "5e07a3fc96dc45268dce6622d2ae9f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e07a3fc96dc45268dce6622d2ae9f93" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38987 } } }
I20260924 13:38:09.016462 31957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:09.016563 31962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:09.017042 31962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:09.018764 31962 catalog_manager.cc:1347] Generated new cluster ID: 115091d2c5b3499fb4326df3718c176f
I20260924 13:38:09.018826 31962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:09.019688 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:09.012595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.25.240.126:38987)
0924 13:38:09.012841 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:09.012850 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:09.012884 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:38:09.013996 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:09.014004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:09.014093 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:09.014249 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:09.014483 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.016309 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.017482 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.018099 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.018227 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:09.018953 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:09.018957 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:09.018957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:09.019163 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:09.019209 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:09.019223 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:09.019305 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:09.019395 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 13:38:09.019475 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260924 13:38:09.022490 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:38:09.041539 31962 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:09.042068 31962 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:09.046887 31962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e07a3fc96dc45268dce6622d2ae9f93: Generated new TSK 0
I20260924 13:38:09.047065 31962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:38:09.059319 31954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:09.055281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.25.240.126:38987)
0924 13:38:09.055341 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:09.055343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:09.055434 (+ 91us) server_negotiation.cc:407] Connection header received
0924 13:38:09.055490 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:09.055491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:09.055503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:09.055583 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:38:09.055708 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.056475 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.057235 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.057592 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.057663 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:09.057825 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:38:09.059098 (+ 1273us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 13:38:09.059232 (+ 134us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 13:38:09.059377 31954 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39450: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 13:38:09.061242 31979 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:09.055192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:38987 (local address 127.0.0.1:39450)
0924 13:38:09.055364 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:09.055374 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:09.055436 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:09.055589 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:09.055590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:09.055592 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:38:09.055642 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:09.055645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.056553 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.056554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:09.057149 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:09.057152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.057681 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.057682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:09.057724 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:09.057729 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:38:09.059198 (+ 1469us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 13:38:09.061148 (+ 1950us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.240.126:38987: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39253/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":1750,"thread_start_us":113,"threads_started":1}
I20260924 13:38:09.061592 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 31883
E20260924 13:38:09.070171 31980 webserver.cc:533] Webserver: sslize failed:
E20260924 13:38:09.070245 31980 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T13:38:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T13:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:09Z Disabled control of system clock
I20260924 13:38:09.106274 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38577
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:38577
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/testchainca.cert with env {}
W20260924 13:38:09.186506 31984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:09.186721 31984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:09.186744 31984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:09.187870 31984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:09.187906 31984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:09.187918 31984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:09.187932 31984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:09.187943 31984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 13:38:09.187955 31984 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 13:38:09.187966 31984 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 13:38:09.189400 31984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:38577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:09.189661 31984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:09.189880 31984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:09.194646 31990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:09.194667 31992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:09.194643 31989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:09.194835 31984 server_base.cc:1034] running on GCE node
I20260924 13:38:09.195218 31984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:09.195501 31984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:09.196686 31984 hybrid_clock.cc:648] HybridClock initialized: now 1790257089196679 us; error 40 us; skew 500 ppm
I20260924 13:38:09.197917 31984 webserver.cc:460] Webserver started at http://127.25.240.126:38539/ using document root <none> and password file <none>
I20260924 13:38:09.198169 31984 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:09.198239 31984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:09.198393 31984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:09.199496 31984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "51a85c26a98f4ef99a8293c2fefb04b4"
format_stamp: "Formatted at 2026-09-24 13:38:09 on dist-test-slave-m5z1"
server_key: "0ff9fa916873ce00f186a3c6d7e08827"
server_key_iv: "80341ab2a8b7738a3be8b1da3e9d27a6"
server_key_version: "encryptionkey@0"
I20260924 13:38:09.199872 31984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "51a85c26a98f4ef99a8293c2fefb04b4"
format_stamp: "Formatted at 2026-09-24 13:38:09 on dist-test-slave-m5z1"
server_key: "0ff9fa916873ce00f186a3c6d7e08827"
server_key_iv: "80341ab2a8b7738a3be8b1da3e9d27a6"
server_key_version: "encryptionkey@0"
I20260924 13:38:09.201166 31984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:09.201992 31998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:09.202167 31984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:09.202273 31984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "51a85c26a98f4ef99a8293c2fefb04b4"
format_stamp: "Formatted at 2026-09-24 13:38:09 on dist-test-slave-m5z1"
server_key: "0ff9fa916873ce00f186a3c6d7e08827"
server_key_iv: "80341ab2a8b7738a3be8b1da3e9d27a6"
server_key_version: "encryptionkey@0"
I20260924 13:38:09.202348 31984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:09.212997 31984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:09.213302 31984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:09.213461 31984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:09.217178 31984 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:38577
I20260924 13:38:09.217222 32050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:38577 every 8 connection(s)
I20260924 13:38:09.217577 31984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:09.218564 32051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:09.220301 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 31984
I20260924 13:38:09.220403 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:09.220577 26561 external_mini_cluster.cc:1428] Setting key 25d3d0bb4259e42adbac89ecfdcaa20d
I20260924 13:38:09.221338 32051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4: Bootstrap starting.
I20260924 13:38:09.222074 32051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:09.222416 32051 log.cc:824] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:09.223160 32051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4: No bootstrap required, opened a new log
I20260924 13:38:09.225342 32051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a85c26a98f4ef99a8293c2fefb04b4" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38577 } }
I20260924 13:38:09.225551 32051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:09.225625 32051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51a85c26a98f4ef99a8293c2fefb04b4, State: Initialized, Role: FOLLOWER
I20260924 13:38:09.225751 32051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a85c26a98f4ef99a8293c2fefb04b4" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38577 } }
I20260924 13:38:09.225855 32051 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:09.225907 32051 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:09.225963 32051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:09.226696 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:09.221092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.25.240.126:38577)
0924 13:38:09.221316 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:09.221325 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:09.221359 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:38:09.221978 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:09.221987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:09.222069 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:09.222231 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:09.222419 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.223708 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.224464 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.225182 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.225305 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:09.225985 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:09.225989 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:09.225989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:09.226195 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:09.226239 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:09.226253 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:09.226323 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:09.226386 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 13:38:09.226465 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260924 13:38:09.226701 32051 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a85c26a98f4ef99a8293c2fefb04b4" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38577 } }
I20260924 13:38:09.226862 32051 leader_election.cc:302] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51a85c26a98f4ef99a8293c2fefb04b4; no voters:
I20260924 13:38:09.227028 32051 leader_election.cc:288] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:09.227063 32056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:09.227239 32056 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [term 1 LEADER]: Becoming Leader. State: Replica: 51a85c26a98f4ef99a8293c2fefb04b4, State: Running, Role: LEADER
I20260924 13:38:09.227289 32051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:09.227373 32056 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a85c26a98f4ef99a8293c2fefb04b4" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38577 } }
I20260924 13:38:09.227715 32056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51a85c26a98f4ef99a8293c2fefb04b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a85c26a98f4ef99a8293c2fefb04b4" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38577 } } }
I20260924 13:38:09.227846 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:38:09.227842 32056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:09.227959 32057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51a85c26a98f4ef99a8293c2fefb04b4. Latest consensus state: current_term: 1 leader_uuid: "51a85c26a98f4ef99a8293c2fefb04b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51a85c26a98f4ef99a8293c2fefb04b4" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38577 } } }
I20260924 13:38:09.228052 32057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4 [sys.catalog]: This master's current role is: LEADER
W20260924 13:38:09.229175 32071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:38:09.229260 32071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:38:09.229359 32064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:09.229511 32064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:09.231004 32064 catalog_manager.cc:1347] Generated new cluster ID: 3b479d5e541e466e892dab1e75920482
I20260924 13:38:09.231060 32064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:09.236987 32064 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:09.237484 32064 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:09.246084 32064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51a85c26a98f4ef99a8293c2fefb04b4: Generated new TSK 0
I20260924 13:38:09.246277 32064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:38:09.453838 32054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:09.253421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.25.240.126:38577)
0924 13:38:09.253489 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:09.253492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:09.253622 (+ 130us) server_negotiation.cc:407] Connection header received
0924 13:38:09.253709 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:09.253710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:09.253722 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:09.253810 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0924 13:38:09.253949 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.254454 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.255241 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.255462 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.255540 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:09.255715 (+ 175us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 13:38:09.453519 (+197804us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 13:38:09.453747 (+ 228us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42430: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 13:38:09.453928 32054 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42430: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 13:38:09.453929 32080 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:09.253353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:38577 (local address 127.0.0.1:42430)
0924 13:38:09.253540 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:09.253549 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:09.253639 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:09.253809 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:09.253810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:09.253812 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 13:38:09.253870 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:09.253873 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.254540 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.254541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:09.255151 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:09.255153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:09.255544 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:09.255546 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:09.255605 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:09.255611 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 13:38:09.453694 (+198083us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 13:38:09.453822 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.240.126:38577: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":105,"threads_started":1}
I20260924 13:38:09.454213 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 31984
2026-09-24T13:38:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (372 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:09 dist-test-slave-m5z1 krb5kdc[32092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:09 dist-test-slave-m5z1 krb5kdc[32092](info): set up 2 sockets
Sep 24 13:38:09 dist-test-slave-m5z1 krb5kdc[32092](info): commencing operation
krb5kdc: starting...
W20260924 13:38:10.687609 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:10 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:10.804056 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39359
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:39359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:10.883761 32105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:10.883962 32105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:10.883985 32105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:10.885109 32105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:10.885142 32105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:10.885156 32105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:10.885169 32105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:10.885180 32105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:10.886583 32105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:39359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:10.886766 32105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:10.886931 32105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:10.891757 32110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:10.891786 32111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:10.891819 32105 server_base.cc:1034] running on GCE node
W20260924 13:38:10.891786 32113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:10.892246 32105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:10.892532 32105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:10.893623 32105 hybrid_clock.cc:648] HybridClock initialized: now 1790257090893615 us; error 34 us; skew 500 ppm
Sep 24 13:38:10 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257090, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:10.896081 32105 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:10.896509 32105 webserver.cc:460] Webserver started at http://127.25.240.126:36725/ using document root <none> and password file <none>
I20260924 13:38:10.896705 32105 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:10.896745 32105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:10.896831 32105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:10.897945 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc"
format_stamp: "Formatted at 2026-09-24 13:38:10 on dist-test-slave-m5z1"
server_key: "2360802de90b95713344787bd79fa6ef"
server_key_iv: "e1574c5894ecc89610da0565edee3979"
server_key_version: "encryptionkey@0"
I20260924 13:38:10.898257 32105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc"
format_stamp: "Formatted at 2026-09-24 13:38:10 on dist-test-slave-m5z1"
server_key: "2360802de90b95713344787bd79fa6ef"
server_key_iv: "e1574c5894ecc89610da0565edee3979"
server_key_version: "encryptionkey@0"
I20260924 13:38:10.899533 32105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:10.900229 32120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:10.900408 32105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:10.900508 32105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc"
format_stamp: "Formatted at 2026-09-24 13:38:10 on dist-test-slave-m5z1"
server_key: "2360802de90b95713344787bd79fa6ef"
server_key_iv: "e1574c5894ecc89610da0565edee3979"
server_key_version: "encryptionkey@0"
I20260924 13:38:10.900590 32105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:10.921515 32105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:10.923844 32105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:10.924009 32105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:10.927980 32105 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:39359
I20260924 13:38:10.928016 32172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:39359 every 8 connection(s)
I20260924 13:38:10.928355 32105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:10.928750 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 32105
I20260924 13:38:10.928856 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:10.929121 26561 external_mini_cluster.cc:1428] Setting key 094aaa07c321bf5b196e5251fdb58cc5
I20260924 13:38:10.929528 32173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:10.932786 32173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc: Bootstrap starting.
I20260924 13:38:10.933430 32173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:10.933782 32173 log.cc:824] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:10.934497 32173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc: No bootstrap required, opened a new log
Sep 24 13:38:10 dist-test-slave-m5z1 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:10.936789 32173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39359 } }
I20260924 13:38:10.936942 32173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:10.936965 32173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c1e5cde31741a0a7fa5a2eeb3d78dc, State: Initialized, Role: FOLLOWER
I20260924 13:38:10.937098 32173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39359 } }
I20260924 13:38:10.937175 32173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:10.937196 32173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:10.937243 32173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:10.937960 32173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39359 } }
I20260924 13:38:10.938073 32173 leader_election.cc:302] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c1e5cde31741a0a7fa5a2eeb3d78dc; no voters:
I20260924 13:38:10.938237 32173 leader_election.cc:288] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:10.938310 32178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:10.938452 32178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [term 1 LEADER]: Becoming Leader. State: Replica: 77c1e5cde31741a0a7fa5a2eeb3d78dc, State: Running, Role: LEADER
I20260924 13:38:10.938494 32173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:10.938532 32178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39359 } }
I20260924 13:38:10.938939 32180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c1e5cde31741a0a7fa5a2eeb3d78dc. Latest consensus state: current_term: 1 leader_uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39359 } } }
I20260924 13:38:10.938946 32179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c1e5cde31741a0a7fa5a2eeb3d78dc" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39359 } } }
I20260924 13:38:10.939110 32180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:10.939121 32179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:10.939486 32187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:10.940014 32187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:10.940531 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:10.929689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35666 (local address 127.25.240.126:39359)
0924 13:38:10.929860 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:10.929867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:10.929901 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:38:10.930305 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:10.930312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:10.930355 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:10.930467 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:10.931217 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:10.932083 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:10.933160 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:10.933658 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:10.933746 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:10.936318 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:10.936322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:10.936322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:10.936332 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:10.938369 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:10.938815 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:10.938818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:10.938818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:10.938926 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:10.939177 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:10.939177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:10.939178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:10.939512 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:10.940111 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:10.940190 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 13:38:10.940322 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260924 13:38:10.941581 32187 catalog_manager.cc:1347] Generated new cluster ID: d1184e9b7aa3456db88941e32e4a2c7c
I20260924 13:38:10.941632 32187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:10.951076 32187 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:10.951566 32187 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:10.958461 32187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c1e5cde31741a0a7fa5a2eeb3d78dc: Generated new TSK 0
I20260924 13:38:10.958664 32187 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:38:10.988852 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39359
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:11.069118 32199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:11.069312 32199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:11.069335 32199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:11.070513 32199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:11.070554 32199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:11.070608 32199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:11.072046 32199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:11.072286 32199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:11.072510 32199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:11.077368 32204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:11.077368 32205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:11.077411 32207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:11.077591 32199 server_base.cc:1034] running on GCE node
I20260924 13:38:11.077886 32199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:11.078127 32199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:11.079286 32199 hybrid_clock.cc:648] HybridClock initialized: now 1790257091079265 us; error 45 us; skew 500 ppm
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.081820 32199 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:38:11.082237 32199 webserver.cc:460] Webserver started at http://127.25.240.65:46729/ using document root <none> and password file <none>
I20260924 13:38:11.082465 32199 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:11.082532 32199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:11.082671 32199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:11.083690 32199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "5b94499072c24210b39f0b6fbf8f41fb"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "ba6d673f1f925ab961721544a2426877"
server_key_iv: "0a49efcfda752ca2c28210bd3aa26108"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.084055 32199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "5b94499072c24210b39f0b6fbf8f41fb"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "ba6d673f1f925ab961721544a2426877"
server_key_iv: "0a49efcfda752ca2c28210bd3aa26108"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.085357 32199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:11.086135 32214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.086297 32199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.086413 32199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "5b94499072c24210b39f0b6fbf8f41fb"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "ba6d673f1f925ab961721544a2426877"
server_key_iv: "0a49efcfda752ca2c28210bd3aa26108"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.086499 32199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:11.105887 32199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:11.108158 32199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:11.108316 32199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:11.108553 32199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:11.108857 32199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:11.108911 32199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.108952 32199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:11.108980 32199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.114750 32199 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:39975
I20260924 13:38:11.114794 32327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:39975 every 8 connection(s)
I20260924 13:38:11.115093 32199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:11.123840 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.115866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:43509 (local address 127.25.240.126:39359)
0924 13:38:11.115912 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.115915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.116074 (+ 159us) server_negotiation.cc:407] Connection header received
0924 13:38:11.116969 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.116970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.116980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.117043 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.117688 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.118179 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.119113 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.119303 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.119352 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.121388 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:11.121389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:11.121389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:11.121394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:11.122658 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.123050 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.123051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.123051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.123101 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.123308 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.123308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.123309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.123426 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:11.123475 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:11.123698 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 13:38:11.123768 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:38:11.124034 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 32199
I20260924 13:38:11.124056 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.115769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39359 (local address 127.25.240.65:43509)
0924 13:38:11.115940 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:11.115982 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:11.116874 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:11.117122 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:11.117126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:11.117364 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:11.117582 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:11.117586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.118254 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.118255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:11.119041 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:11.119043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.119374 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.119375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:11.119479 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.119995 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:11.120003 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:11.121313 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:11.122728 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:11.122734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:11.122735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:11.122983 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:11.123158 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:11.123158 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:11.123158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:11.123255 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:11.123481 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:11.123485 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:11.123592 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 13:38:11.123672 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:38:11.123783 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260924 13:38:11.124157 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:11.124359 26561 external_mini_cluster.cc:1428] Setting key 90474d1535b870934b583f6e8868425d
I20260924 13:38:11.124605 32328 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39359
I20260924 13:38:11.124722 32328 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:11.124912 32328 heartbeater.cc:511] Master 127.25.240.126:39359 requested a full tablet report, sending...
I20260924 13:38:11.125463 32137 ts_manager.cc:194] Registered new tserver with Master: 5b94499072c24210b39f0b6fbf8f41fb (127.25.240.65:39975)
I20260924 13:38:11.126101 32137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:43509
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:38:11.149101 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39359
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:11.229332 32333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:11.229529 32333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:11.229601 32333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:11.230741 32333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:11.230783 32333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:11.230835 32333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:11.232308 32333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:11.232558 32333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:11.232807 32333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:11.237790 32338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:11.237910 32339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:11.238009 32333 server_base.cc:1034] running on GCE node
W20260924 13:38:11.238073 32341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:11.238305 32333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:11.238552 32333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:11.239714 32333 hybrid_clock.cc:648] HybridClock initialized: now 1790257091239711 us; error 34 us; skew 500 ppm
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.242192 32333 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:38:11.242663 32333 webserver.cc:460] Webserver started at http://127.25.240.66:34909/ using document root <none> and password file <none>
I20260924 13:38:11.242908 32333 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:11.242977 32333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:11.243121 32333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:11.244153 32333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "0d8561468a764b77b14be21abcb6003b"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "6ad886cf2e09f64f95a781469e3e7d9c"
server_key_iv: "faabdfcff740b9e327aed26ae53bcc83"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.244521 32333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "0d8561468a764b77b14be21abcb6003b"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "6ad886cf2e09f64f95a781469e3e7d9c"
server_key_iv: "faabdfcff740b9e327aed26ae53bcc83"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.245837 32333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:11.246533 32348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.246693 32333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:11.246798 32333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "0d8561468a764b77b14be21abcb6003b"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "6ad886cf2e09f64f95a781469e3e7d9c"
server_key_iv: "faabdfcff740b9e327aed26ae53bcc83"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.246872 32333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:11.256183 32333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:11.258037 32333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:11.258195 32333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:11.258421 32333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:11.258723 32333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:11.258776 32333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.258817 32333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:11.258845 32333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.264714 32333 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:34609
I20260924 13:38:11.264755 32461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:34609 every 8 connection(s)
I20260924 13:38:11.265072 32333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:11.273874 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 32333
I20260924 13:38:11.273998 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:11.274197 26561 external_mini_cluster.cc:1428] Setting key 40f2ace50423dc65bf8dab6cb41457b6
I20260924 13:38:11.274555 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.265879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:59233 (local address 127.25.240.126:39359)
0924 13:38:11.265937 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.265939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.266149 (+ 210us) server_negotiation.cc:407] Connection header received
0924 13:38:11.267122 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.267123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.267136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.267188 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.267825 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.268337 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.269334 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.269542 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.269682 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.271740 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:11.271742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:11.271742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:11.271750 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:11.272975 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.273381 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.273382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.273383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.273443 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.273761 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.273762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.273762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.273917 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:11.273986 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:11.274351 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 13:38:11.274456 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 13:38:11.274675 32464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.265838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39359 (local address 127.25.240.66:59233)
0924 13:38:11.266032 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:11.266069 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:11.267006 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:11.267270 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:11.267276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:11.267538 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:11.267728 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:11.267732 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.268442 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.268443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:11.269242 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:11.269243 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.269690 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.269690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:11.269818 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.270298 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:11.270306 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:11.271639 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:11.273059 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:11.273065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:11.273068 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:11.273282 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:11.273518 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:11.273519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:11.273519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:11.273694 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:11.274015 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:11.274020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:11.274189 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 13:38:11.274296 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 13:38:11.274465 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260924 13:38:11.275197 32462 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39359
I20260924 13:38:11.275314 32462 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:11.275502 32462 heartbeater.cc:511] Master 127.25.240.126:39359 requested a full tablet report, sending...
I20260924 13:38:11.275915 32137 ts_manager.cc:194] Registered new tserver with Master: 0d8561468a764b77b14be21abcb6003b (127.25.240.66:34609)
I20260924 13:38:11.276412 32137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:59233
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:38:11.299746 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39359
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:11.380935 32467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:11.381131 32467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:11.381153 32467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:11.382361 32467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:11.382408 32467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:11.382436 32467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:11.383889 32467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:11.384138 32467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:11.384374 32467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:11.389464 32473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:11.389505 32472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:11.389514 32467 server_base.cc:1034] running on GCE node
W20260924 13:38:11.389510 32475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:11.390000 32467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:11.390280 32467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:11.391429 32467 hybrid_clock.cc:648] HybridClock initialized: now 1790257091391404 us; error 37 us; skew 500 ppm
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.394156 32467 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:38:11.394546 32467 webserver.cc:460] Webserver started at http://127.25.240.67:33919/ using document root <none> and password file <none>
I20260924 13:38:11.394727 32467 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:11.394767 32467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:11.394858 32467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:11.395869 32467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "1e0cb05014744b02a42fc60dabd20faa"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "18f7cdb083e14e390e39029b48de04a5"
server_key_iv: "cc437bac73c0ef1e58e54a42a9ce9917"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.396178 32467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "1e0cb05014744b02a42fc60dabd20faa"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "18f7cdb083e14e390e39029b48de04a5"
server_key_iv: "cc437bac73c0ef1e58e54a42a9ce9917"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.397434 32467 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:11.398402 32482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.398573 32467 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:11.398655 32467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "1e0cb05014744b02a42fc60dabd20faa"
format_stamp: "Formatted at 2026-09-24 13:38:11 on dist-test-slave-m5z1"
server_key: "18f7cdb083e14e390e39029b48de04a5"
server_key_iv: "cc437bac73c0ef1e58e54a42a9ce9917"
server_key_version: "encryptionkey@0"
I20260924 13:38:11.398711 32467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:11.406639 32467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:11.408381 32467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:11.408488 32467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:11.408653 32467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:11.408893 32467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:11.408923 32467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.408943 32467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:11.408962 32467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:11.414580 32467 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:34137
I20260924 13:38:11.414638 32595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:34137 every 8 connection(s)
I20260924 13:38:11.414935 32467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:11.423591 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.415627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:43331 (local address 127.25.240.126:39359)
0924 13:38:11.415696 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.415698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.415903 (+ 205us) server_negotiation.cc:407] Connection header received
0924 13:38:11.416796 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.416797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.416806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.416874 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.417448 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.418024 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.418970 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.419144 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.419194 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.421222 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:11.421223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:11.421224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:11.421230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:11.422492 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.422833 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.422834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.422834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.422883 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.423094 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.423095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.423095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.423190 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:11.423239 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:11.423469 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 13:38:11.423526 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:38:11.423871 32598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.415613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39359 (local address 127.25.240.67:43331)
0924 13:38:11.415779 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:11.415813 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:11.416708 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:11.416932 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:11.416935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:11.417176 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:11.417362 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:11.417366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.418103 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.418104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:11.418896 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:11.418898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.419214 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.419215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:11.419313 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.419845 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:11.419852 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:11.421142 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:11.422558 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:11.422563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:11.422565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:11.422768 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:11.422939 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:11.422939 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:11.422939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:11.423038 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:11.423244 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:11.423247 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:11.423353 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 13:38:11.423442 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 13:38:11.423559 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":114,"threads_started":1}
I20260924 13:38:11.424371 32596 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39359
I20260924 13:38:11.424439 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 32467
I20260924 13:38:11.424484 32596 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:11.424544 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:11.424675 32596 heartbeater.cc:511] Master 127.25.240.126:39359 requested a full tablet report, sending...
I20260924 13:38:11.424722 26561 external_mini_cluster.cc:1428] Setting key 32dde79aa9cb6413241328b162f42e8f
I20260924 13:38:11.425112 32136 ts_manager.cc:194] Registered new tserver with Master: 1e0cb05014744b02a42fc60dabd20faa (127.25.240.67:34137)
I20260924 13:38:11.425673 32136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:43331
I20260924 13:38:11.425772 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/test-admin.keytab.
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.442039 26561 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 13:38:11.443634 32603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:11.442858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.25.240.65:39975)
0924 13:38:11.443051 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.443056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.443064 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:11.443088 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.443090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.443103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.443187 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.443536 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40152: BlockingRecv error: recv got EOF from 127.0.0.1:40152 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":106,"threads_started":1}
W20260924 13:38:11.443760 26561 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5cc' of type FILE
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.444676 26561 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.445540 26561 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:38:11.452899 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.446073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.25.240.65:39975)
0924 13:38:11.446136 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.446138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.446146 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:11.446192 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.446193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.446202 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.446255 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.446621 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.447321 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.448021 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.448328 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.448398 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.450554 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:11.450561 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:11.450561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:11.450567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:11.451957 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.452261 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.452263 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.452263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.452314 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.452481 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.452481 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.452481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.452599 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:11.452706 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:11.452770 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 13:38:11.452826 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 13:38:11.454627 32603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:11.454156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.25.240.65:39975)
0924 13:38:11.454193 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.454195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.454228 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:38:11.454275 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.454276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.454283 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.454333 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.454554 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40172: BlockingRecv error: recv got EOF from 127.0.0.1:40172 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 13:38:11.454759 26561 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5cc' of type FILE
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.455681 26561 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.456523 26561 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:38:11.462981 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.457047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.25.240.65:39975)
0924 13:38:11.457093 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.457095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.457111 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:38:11.457163 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.457163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.457171 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.457218 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.457548 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.458052 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.458690 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.458910 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.458981 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.460804 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:11.460805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:11.460805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:11.460811 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:11.462191 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.462470 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.462471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.462471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.462518 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.462657 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.462657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.462658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.462729 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:11.462790 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:11.462873 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 13:38:11.462925 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 13:38:11.464361 32603 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:11.463911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.25.240.65:39975)
0924 13:38:11.463947 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.463949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.463982 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:38:11.464038 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.464038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.464045 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.464086 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.464319 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40190: BlockingRecv error: recv got EOF from 127.0.0.1:40190 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 13:38:11.464552 26561 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790257062855231-26561-0/krb5kdc/krb5cc)
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.465437 26561 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:11.466279 26561 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:38:11.472710 32603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:11.466737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.25.240.65:39975)
0924 13:38:11.466772 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:11.466774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:11.466813 (+ 39us) server_negotiation.cc:407] Connection header received
0924 13:38:11.466871 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:11.466872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:11.466879 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:11.466918 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:11.467253 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.467690 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.468312 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:11.468494 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:11.468545 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:11.470500 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:11.470501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:11.470502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:11.470508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:11.471616 (+ 1108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.471976 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.471977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.471977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.472033 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:11.472229 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:11.472229 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:11.472230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:11.472325 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:11.472429 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:11.472537 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 13:38:11.472605 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:11.473340 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 32199
I20260924 13:38:11.478451 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 32333
I20260924 13:38:11.482674 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 32467
I20260924 13:38:11.486743 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 32105
2026-09-24T13:38:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1856 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32616](info): set up 2 sockets
Sep 24 13:38:11 dist-test-slave-m5z1 krb5kdc[32616](info): commencing operation
krb5kdc: starting...
W20260924 13:38:12.510106 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:12 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.126.keytab.
Entry for principal oryx/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.126.keytab.
I20260924 13:38:12.624343 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35077
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.126.keytab
--principal=oryx/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:35077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:12.706995 32629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:12.707211 32629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:12.707247 32629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:12.708489 32629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:12.708545 32629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:12.708568 32629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:12.708598 32629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:12.708626 32629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:12.710109 32629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:35077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.126.keytab
--principal=oryx/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:12.710364 32629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:12.710577 32629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:12.715348 32635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:12.715430 32637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:12.715477 32634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:12.715477 32629 server_base.cc:1034] running on GCE node
I20260924 13:38:12.715734 32629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:12.715960 32629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:12.717094 32629 hybrid_clock.cc:648] HybridClock initialized: now 1790257092717067 us; error 42 us; skew 500 ppm
Sep 24 13:38:12 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:12.728236 32629 init.cc:377] Logged in from keytab as oryx/127.25.240.126@KRBTEST.COM (short username oryx)
I20260924 13:38:12.728708 32629 webserver.cc:460] Webserver started at http://127.25.240.126:40137/ using document root <none> and password file <none>
I20260924 13:38:12.728888 32629 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:12.728928 32629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:12.729015 32629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:12.730047 32629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "250ae62b280043d28245890fa6bbe456"
format_stamp: "Formatted at 2026-09-24 13:38:12 on dist-test-slave-m5z1"
server_key: "57d21d3365b1beb83bee5c3a5c92fc71"
server_key_iv: "d5f9b2fa2d12aa0a8727f846dea23bb7"
server_key_version: "encryptionkey@0"
I20260924 13:38:12.730353 32629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "250ae62b280043d28245890fa6bbe456"
format_stamp: "Formatted at 2026-09-24 13:38:12 on dist-test-slave-m5z1"
server_key: "57d21d3365b1beb83bee5c3a5c92fc71"
server_key_iv: "d5f9b2fa2d12aa0a8727f846dea23bb7"
server_key_version: "encryptionkey@0"
I20260924 13:38:12.731566 32629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:12.732244 32644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:12.732445 32629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:12.732551 32629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "250ae62b280043d28245890fa6bbe456"
format_stamp: "Formatted at 2026-09-24 13:38:12 on dist-test-slave-m5z1"
server_key: "57d21d3365b1beb83bee5c3a5c92fc71"
server_key_iv: "d5f9b2fa2d12aa0a8727f846dea23bb7"
server_key_version: "encryptionkey@0"
I20260924 13:38:12.732618 32629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:12.742363 32629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:12.744089 32629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:12.744236 32629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:12.747994 32629 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:35077
I20260924 13:38:12.748059 32696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:35077 every 8 connection(s)
I20260924 13:38:12.748396 32629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:12.748900 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 32629
I20260924 13:38:12.748980 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:12.749207 26561 external_mini_cluster.cc:1428] Setting key 7df837194f9b949211c4761076b8d65b
I20260924 13:38:12.749492 32697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:12.752223 32697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: Bootstrap starting.
I20260924 13:38:12.752843 32697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:12.753131 32697 log.cc:824] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:12.753829 32697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: No bootstrap required, opened a new log
Sep 24 13:38:12 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:12.755980 32697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } }
I20260924 13:38:12.756105 32697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:12.756129 32697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 250ae62b280043d28245890fa6bbe456, State: Initialized, Role: FOLLOWER
I20260924 13:38:12.756270 32697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } }
I20260924 13:38:12.756346 32697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:12.756368 32697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:12.756418 32697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:12.757084 32697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } }
I20260924 13:38:12.757212 32697 leader_election.cc:302] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 250ae62b280043d28245890fa6bbe456; no voters:
I20260924 13:38:12.757413 32697 leader_election.cc:288] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:12.757440 32702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:12.757546 32702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 1 LEADER]: Becoming Leader. State: Replica: 250ae62b280043d28245890fa6bbe456, State: Running, Role: LEADER
I20260924 13:38:12.757664 32702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } }
I20260924 13:38:12.757716 32697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:12.757982 32702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } }
I20260924 13:38:12.758129 32702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:12.758313 32702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 250ae62b280043d28245890fa6bbe456. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } }
I20260924 13:38:12.758436 32702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:12.758735 32709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:12.759332 32709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:12.760437 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:12.749759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38650 (local address 127.25.240.126:35077)
0924 13:38:12.749952 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:12.749967 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:12.749991 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:38:12.750610 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:12.750618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:12.750666 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:12.750822 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:12.751465 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:12.752371 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:12.753215 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:12.753691 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:12.753804 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:12.756378 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:12.756382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:12.756382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:12.756395 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:12.758361 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:12.758799 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:12.758802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:12.758803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:12.758910 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:12.759130 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:12.759130 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:12.759131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:12.759440 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:12.759681 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:12.760113 (+ 432us) server_negotiation.cc:299] Negotiation successful
0924 13:38:12.760246 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260924 13:38:12.761951 32709 catalog_manager.cc:1347] Generated new cluster ID: 907e9bca243a4847b893d33ae34dd4b7
I20260924 13:38:12.762009 32709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:12.772583 32709 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:12.773075 32709 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:12.777855 32709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: Generated new TSK 0
I20260924 13:38:12.778040 32709 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab.
Entry for principal oryx/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab.
I20260924 13:38:12.806527 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35077
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab
--principal=oryx/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:12.886310 32723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:12.886543 32723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:12.886580 32723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:12.887712 32723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:12.887782 32723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:12.887827 32723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:12.889286 32723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab
--principal=oryx/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:12.889588 32723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:12.889818 32723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:12.894891 32731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:12.894907 32729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:12.894985 32723 server_base.cc:1034] running on GCE node
W20260924 13:38:12.895066 32728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:12.895313 32723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:12.895570 32723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:12.896732 32723 hybrid_clock.cc:648] HybridClock initialized: now 1790257092896727 us; error 37 us; skew 500 ppm
Sep 24 13:38:12 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:12.899219 32723 init.cc:377] Logged in from keytab as oryx/127.25.240.65@KRBTEST.COM (short username oryx)
I20260924 13:38:12.899715 32723 webserver.cc:460] Webserver started at http://127.25.240.65:38753/ using document root <none> and password file <none>
I20260924 13:38:12.899962 32723 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:12.900030 32723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:12.900163 32723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:12.901189 32723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "d2135c02dfb042069cfb5a9747cbff71"
format_stamp: "Formatted at 2026-09-24 13:38:12 on dist-test-slave-m5z1"
server_key: "860fa0bd286c5fe213262136344091ca"
server_key_iv: "7adb49493adfd3a53fcdcc58a7ae678e"
server_key_version: "encryptionkey@0"
I20260924 13:38:12.901602 32723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "d2135c02dfb042069cfb5a9747cbff71"
format_stamp: "Formatted at 2026-09-24 13:38:12 on dist-test-slave-m5z1"
server_key: "860fa0bd286c5fe213262136344091ca"
server_key_iv: "7adb49493adfd3a53fcdcc58a7ae678e"
server_key_version: "encryptionkey@0"
I20260924 13:38:12.902948 32723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:38:12.903651 32738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:12.903810 32723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:12.903908 32723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "d2135c02dfb042069cfb5a9747cbff71"
format_stamp: "Formatted at 2026-09-24 13:38:12 on dist-test-slave-m5z1"
server_key: "860fa0bd286c5fe213262136344091ca"
server_key_iv: "7adb49493adfd3a53fcdcc58a7ae678e"
server_key_version: "encryptionkey@0"
I20260924 13:38:12.903997 32723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:12.957263 32723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:12.959553 32723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:12.959715 32723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:12.959967 32723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:12.960301 32723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:12.960356 32723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:12.960399 32723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:12.960423 32723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:12.966286 32723 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:43813
I20260924 13:38:12.966336 383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:43813 every 8 connection(s)
I20260924 13:38:12.966650 32723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:38:12.971750 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 32723
I20260924 13:38:12.971879 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:12.972110 26561 external_mini_cluster.cc:1428] Setting key ac258a97024675c8390c0b1c1e6abbe0
Sep 24 13:38:12 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:12.975941 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:12.967355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:54779 (local address 127.25.240.126:35077)
0924 13:38:12.967419 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:12.967421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:12.967644 (+ 223us) server_negotiation.cc:407] Connection header received
0924 13:38:12.968133 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:12.968134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:12.968143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:12.968202 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:12.969117 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:12.969635 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:12.970576 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:12.970775 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:12.970829 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:12.973072 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:12.973073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:12.973074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:12.973081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:12.974540 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:12.974961 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:12.974962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:12.974962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:12.975025 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:12.975288 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:12.975289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:12.975290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:12.975410 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:12.975483 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:12.975761 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 13:38:12.975848 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:38:12.976147 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:12.967319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.65:54779)
0924 13:38:12.967507 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:12.967549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:12.968046 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:12.968271 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:12.968274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:12.968829 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:12.969025 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:12.969030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:12.969714 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:12.969714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:12.970501 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:12.970503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:12.970848 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:12.970849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:12.970948 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:12.971377 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:12.971384 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:12.972969 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:12.974644 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:12.974651 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:12.974653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:12.974884 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:12.975093 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:12.975094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:12.975094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:12.975220 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:12.975485 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:12.975492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:12.975631 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 13:38:12.975739 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 13:38:12.975906 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260924 13:38:12.976653 384 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35077
I20260924 13:38:12.976765 384 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:12.976984 384 heartbeater.cc:511] Master 127.25.240.126:35077 requested a full tablet report, sending...
I20260924 13:38:12.977595 32661 ts_manager.cc:194] Registered new tserver with Master: d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813)
I20260924 13:38:12.978561 32661 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:54779
WARNING: no policy specified for oryx/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab.
Entry for principal oryx/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab.
I20260924 13:38:12.997175 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35077
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab
--principal=oryx/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:13.076422 389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:13.076670 389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:13.076709 389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:13.077889 389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:13.077960 389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:13.078001 389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:13.079425 389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab
--principal=oryx/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:13.079694 389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:13.079931 389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:13.084800 396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.084934 389 server_base.cc:1034] running on GCE node
W20260924 13:38:13.084878 395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.084803 398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.085261 389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:13.085489 389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:13.086644 389 hybrid_clock.cc:648] HybridClock initialized: now 1790257093086625 us; error 31 us; skew 500 ppm
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:13.088989 389 init.cc:377] Logged in from keytab as oryx/127.25.240.66@KRBTEST.COM (short username oryx)
I20260924 13:38:13.089396 389 webserver.cc:460] Webserver started at http://127.25.240.66:33629/ using document root <none> and password file <none>
I20260924 13:38:13.089660 389 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.089730 389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.089880 389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:13.090886 389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "0dd242ac904746a9a95f1fda67689068"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "5a64223ac4c6b5213efbdb259cfd5954"
server_key_iv: "a2da9317b41d857a795a11d2b1cb6966"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.091255 389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "0dd242ac904746a9a95f1fda67689068"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "5a64223ac4c6b5213efbdb259cfd5954"
server_key_iv: "a2da9317b41d857a795a11d2b1cb6966"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.092540 389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:13.093243 405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.093422 389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:13.093518 389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "0dd242ac904746a9a95f1fda67689068"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "5a64223ac4c6b5213efbdb259cfd5954"
server_key_iv: "a2da9317b41d857a795a11d2b1cb6966"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.093609 389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.108156 389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:13.109982 389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:13.110131 389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:13.110345 389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:13.110642 389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:13.110694 389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.110738 389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:13.110769 389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.116459 389 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:39671
I20260924 13:38:13.116497 518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:39671 every 8 connection(s)
I20260924 13:38:13.116830 389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:38:13.121747 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 389
I20260924 13:38:13.121850 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:13.122041 26561 external_mini_cluster.cc:1428] Setting key 704e0810eeec9f0b14d1f10fb6d7737e
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:13.126219 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.117587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:46827 (local address 127.25.240.126:35077)
0924 13:38:13.117629 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.117631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.117814 (+ 183us) server_negotiation.cc:407] Connection header received
0924 13:38:13.118274 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.118275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.118284 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.118340 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.119190 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.119673 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.120585 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.120766 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.120822 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.123144 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.123146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.123146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.123154 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.124766 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.125194 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.125195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.125195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.125247 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.125607 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.125608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.125608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.125709 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.125762 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.126081 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.126155 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:38:13.126510 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.117520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.66:46827)
0924 13:38:13.117689 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.117729 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.118196 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.118389 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.118392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.118938 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.119117 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.119121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.119747 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.119748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.120509 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.120510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.120858 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.120859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.120962 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.121369 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.121376 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.123048 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.124853 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.124857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.124859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.125077 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.125357 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.125357 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.125358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.125499 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.125774 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.125779 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.125949 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.126061 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.126232 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":110,"threads_started":1}
I20260924 13:38:13.127017 519 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35077
I20260924 13:38:13.127126 519 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:13.127318 519 heartbeater.cc:511] Master 127.25.240.126:35077 requested a full tablet report, sending...
I20260924 13:38:13.127723 32661 ts_manager.cc:194] Registered new tserver with Master: 0dd242ac904746a9a95f1fda67689068 (127.25.240.66:39671)
I20260924 13:38:13.128283 32661 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:46827
WARNING: no policy specified for oryx/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab.
Entry for principal oryx/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab.
I20260924 13:38:13.148291 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35077
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab
--principal=oryx/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:13.228158 524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:13.228348 524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:13.228370 524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:13.229431 524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:13.229462 524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:13.229488 524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:13.230836 524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab
--principal=oryx/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:13.231004 524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:13.231163 524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:13.235934 532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.235940 530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.235973 529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.236248 524 server_base.cc:1034] running on GCE node
I20260924 13:38:13.236462 524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:13.236685 524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:13.237838 524 hybrid_clock.cc:648] HybridClock initialized: now 1790257093237822 us; error 33 us; skew 500 ppm
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:13.240119 524 init.cc:377] Logged in from keytab as oryx/127.25.240.67@KRBTEST.COM (short username oryx)
I20260924 13:38:13.240558 524 webserver.cc:460] Webserver started at http://127.25.240.67:33711/ using document root <none> and password file <none>
I20260924 13:38:13.240772 524 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.240837 524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.240988 524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:13.242043 524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "45fda73093f549059ff1d2f8ef9e411e"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "56b8b95c4464ba002ff42fd2badf94bb"
server_key_iv: "7399842ec2d842b5165530f6bbd68845"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.242396 524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "45fda73093f549059ff1d2f8ef9e411e"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "56b8b95c4464ba002ff42fd2badf94bb"
server_key_iv: "7399842ec2d842b5165530f6bbd68845"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.243700 524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:13.244406 539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.244572 524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:13.244668 524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "45fda73093f549059ff1d2f8ef9e411e"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "56b8b95c4464ba002ff42fd2badf94bb"
server_key_iv: "7399842ec2d842b5165530f6bbd68845"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.244745 524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.251765 524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:13.253685 524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:13.253836 524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:13.254052 524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:13.254346 524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:13.254400 524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.254442 524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:13.254468 524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.260532 524 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:36131
I20260924 13:38:13.260601 652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:36131 every 8 connection(s)
I20260924 13:38:13.260936 524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:38:13.262771 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 524
I20260924 13:38:13.262876 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:13.263087 26561 external_mini_cluster.cc:1428] Setting key 7c9293766e4e902a05de05f890f5be91
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:13.270025 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.261664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:50099 (local address 127.25.240.126:35077)
0924 13:38:13.261735 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.261737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.261954 (+ 217us) server_negotiation.cc:407] Connection header received
0924 13:38:13.262400 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.262401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.262411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.262464 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.263721 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.264235 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.265240 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.265451 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.265535 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.267671 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.267672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.267673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.267679 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.268839 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.269192 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.269193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.269193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.269241 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.269443 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.269444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.269444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.269540 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.269658 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.269902 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.269961 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:38:13.270229 655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.261649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.67:50099)
0924 13:38:13.261838 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.261872 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.262321 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.262515 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.262518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.263354 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.263645 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.263652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.264331 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.264332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.265131 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.265133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.265545 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.265546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.265701 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.266226 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.266234 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.267585 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.268907 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.268912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.268913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.269128 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.269298 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.269298 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.269299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.269391 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.269674 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.269678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.269785 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.269870 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.269991 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
I20260924 13:38:13.270732 653 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35077
I20260924 13:38:13.270810 653 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:13.270953 653 heartbeater.cc:511] Master 127.25.240.126:35077 requested a full tablet report, sending...
I20260924 13:38:13.271292 32661 ts_manager.cc:194] Registered new tserver with Master: 45fda73093f549059ff1d2f8ef9e411e (127.25.240.67:36131)
I20260924 13:38:13.271621 32661 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:50099
I20260924 13:38:13.274716 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.240.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.125.keytab.
Entry for principal oryx/127.25.240.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.125.keytab.
I20260924 13:38:13.299795 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.125:46219
--webserver_interface=127.25.240.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.125.keytab
--principal=oryx/127.25.240.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:35077,127.25.240.125:46219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:13.379487 658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:13.379686 658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:13.379709 658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:13.380841 658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:13.380891 658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:13.380906 658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:13.380944 658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:13.380973 658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:13.382506 658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:35077,127.25.240.125:46219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.125.keytab
--principal=oryx/127.25.240.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.125:46219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.240.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
W20260924 13:38:13.382632 658 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.240.126:35077,127.25.240.125:46219'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 13:38:13.382774 658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:13.382998 658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:13.387810 666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.387821 664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.387907 663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.387956 658 server_base.cc:1034] running on GCE node
I20260924 13:38:13.388238 658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:13.388477 658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:13.389640 658 hybrid_clock.cc:648] HybridClock initialized: now 1790257093389602 us; error 37 us; skew 500 ppm
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:13.392063 658 init.cc:377] Logged in from keytab as oryx/127.25.240.125@KRBTEST.COM (short username oryx)
I20260924 13:38:13.392530 658 webserver.cc:460] Webserver started at http://127.25.240.125:42785/ using document root <none> and password file <none>
I20260924 13:38:13.392812 658 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.392889 658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.393044 658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:13.394207 658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data/instance:
uuid: "b92a645f83a2488b9555d7c02550f35c"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "54b471efd8fe825d866c957d0c532579"
server_key_iv: "64d05001aed02678ca9e46da4bb97aef"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.394593 658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal/instance:
uuid: "b92a645f83a2488b9555d7c02550f35c"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "54b471efd8fe825d866c957d0c532579"
server_key_iv: "64d05001aed02678ca9e46da4bb97aef"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.395900 658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:13.396647 673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.396814 658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:13.396914 658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
uuid: "b92a645f83a2488b9555d7c02550f35c"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "54b471efd8fe825d866c957d0c532579"
server_key_iv: "64d05001aed02678ca9e46da4bb97aef"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.396989 658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.413203 658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:13.415124 658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:13.415279 658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:13.419205 658 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.125:46219
I20260924 13:38:13.419237 726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.125:46219 every 8 connection(s)
I20260924 13:38:13.419584 658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
I20260924 13:38:13.420540 727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:13.422015 727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:38:13.424441 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 658
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.125@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.240.125@KRBTEST.COM
I20260924 13:38:13.431475 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.422682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.25.240.126:35077)
0924 13:38:13.422754 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.422756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.422944 (+ 188us) server_negotiation.cc:407] Connection header received
0924 13:38:13.423378 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.423380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.423391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.423444 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.424005 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.424671 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.426028 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.426212 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.426260 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.428589 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.428590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.428590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.428597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.430078 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.430494 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.430495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.430495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.430556 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.430800 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.430801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.430801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.430961 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.431037 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.431295 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.431384 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:38:13.431644 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.422654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:38662)
0924 13:38:13.422828 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.422862 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.423298 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.423495 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.423498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.423736 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.423925 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.423928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.424783 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.424784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.425944 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.425947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.426286 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.426287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.426416 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.426922 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.426932 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.428497 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.430157 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.430164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.430166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.430409 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.430628 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.430629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.430629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.430730 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.431038 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.431044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.431168 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.431275 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.431421 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260924 13:38:13.432284 727 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:38:13.433370 731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.424965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56790 (local address 127.25.240.125:46219)
0924 13:38:13.425149 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.425154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.425173 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:38:13.425196 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.425197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.425217 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.425296 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.425913 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.426869 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.427605 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.427879 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.427941 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.430209 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.430215 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.430215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.430221 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.432031 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.432365 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.432369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.432369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.432428 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.432773 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.432774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.432774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.432948 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.433108 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.433177 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.433250 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":115,"threads_started":1}
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.125@KRBTEST.COM for oryx/127.25.240.125@KRBTEST.COM
I20260924 13:38:13.439443 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.432710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.0.0.1:56798)
0924 13:38:13.432757 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.432769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.432803 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.433082 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.433083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.433207 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.433248 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.433249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.434009 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.434010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.434475 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.434476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.434883 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.434884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.434959 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.435348 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.435352 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.436582 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.438702 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.438702 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.438703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.438851 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.439022 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.439023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.439023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.439070 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.439237 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.439238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.439268 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.439316 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.439386 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:38:13.439625 732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.432796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56798 (local address 127.25.240.125:46219)
0924 13:38:13.432922 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.432923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.432931 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:13.432987 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.432988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.432995 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.433082 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.433321 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.433882 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.434570 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.434785 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.434877 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.436652 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.436653 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.436653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.436659 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.438633 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.438916 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.438916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.438917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.438967 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.439107 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.439108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.439108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.439190 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.439235 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.439330 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.439429 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260924 13:38:13.440857 727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Bootstrap starting.
I20260924 13:38:13.441428 727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:13.441731 727 log.cc:824] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:13.442382 727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: No bootstrap required, opened a new log
I20260924 13:38:13.443681 727 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } }
I20260924 13:38:13.443795 727 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:13.443817 727 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92a645f83a2488b9555d7c02550f35c, State: Initialized, Role: FOLLOWER
I20260924 13:38:13.443900 727 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } }
I20260924 13:38:13.444123 734 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } } }
I20260924 13:38:13.444188 727 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:13.444204 734 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:38:13.445802 745 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:38:13.445858 745 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:38:13.447511 658 master_runner.cc:428] Detected that this master b92a645f83a2488b9555d7c02550f35c is joining an existing cluster
I20260924 13:38:13.447559 658 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.240.125:46219
I20260924 13:38:13.454748 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.448595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.0.0.1:56800)
0924 13:38:13.448730 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.448741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.448830 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.448955 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.448956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.449147 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.449218 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.449219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.449998 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.449999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.450563 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.450564 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.450971 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.450972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.451047 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.451495 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.451500 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.451800 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.453653 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.453654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.453654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.453842 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.454054 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.454055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.454055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.454110 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.454385 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.454386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.454438 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.454532 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.454633 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260924 13:38:13.455107 732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.448730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56800 (local address 127.25.240.125:46219)
0924 13:38:13.448764 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.448765 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.448825 (+ 60us) server_negotiation.cc:407] Connection header received
0924 13:38:13.448890 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.448891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.448898 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.448939 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.449289 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.449902 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.450662 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.450870 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.450958 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.451897 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.451898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.451898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.451905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.453524 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.453930 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.453931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.453931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.453982 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.454178 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.454179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.454179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.454295 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.454353 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.454555 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.455027 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":16}
I20260924 13:38:13.456454 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.449771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:38666)
0924 13:38:13.449924 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.449933 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.450009 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.450183 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.450184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.450365 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.450417 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.450418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.451374 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.451374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.452235 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.452237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.452778 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.452779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.452853 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.453228 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.453232 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.453808 (+ 576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.455606 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.455607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.455607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.455760 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.455988 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.455988 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.455989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.456028 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.456243 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.456243 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.456270 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.456325 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.456393 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260924 13:38:13.456523 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.449813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38666 (local address 127.25.240.126:35077)
0924 13:38:13.449850 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.449851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.449985 (+ 134us) server_negotiation.cc:407] Connection header received
0924 13:38:13.450081 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.450082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.450093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.450123 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.450534 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.451277 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.452336 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.452647 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.452688 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.453909 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.453910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.453911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.453918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.455521 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.455836 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.455837 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.455838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.455912 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.456067 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.456068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.456068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.456186 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.456254 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.456350 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.456425 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.126@KRBTEST.COM for oryx/127.25.240.125@KRBTEST.COM
I20260924 13:38:13.465842 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.458683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.0.0.1:56810)
0924 13:38:13.458858 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.458924 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.459010 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.459164 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.459166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.459473 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.459525 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.459527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.460361 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.460362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.460917 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.460919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.461354 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.461355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.461417 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.461849 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.461853 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.463098 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.464796 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.464802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.464804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.464991 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.465204 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.465205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.465206 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.465257 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.465509 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.465515 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.465612 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.465691 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.465776 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260924 13:38:13.465890 732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.458754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.25.240.125:46219)
0924 13:38:13.458803 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.458805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.458992 (+ 187us) server_negotiation.cc:407] Connection header received
0924 13:38:13.459088 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.459089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.459098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.459158 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.459594 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.460280 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.460991 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.461281 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.461348 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.463176 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.463177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.463177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.463184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.464716 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.465064 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.465065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.465065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.465136 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.465323 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.465324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.465324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.465449 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.465515 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.465734 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.465805 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 13:38:13.466420 32661 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.240.125:46219: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b92a645f83a2488b9555d7c02550f35c"
member_type: NON_VOTER
last_known_addr {
host: "127.25.240.125"
port: 46219
}
attrs {
promote: true
}
}
dest_uuid: "250ae62b280043d28245890fa6bbe456"
cas_config_opid_index: -1
I20260924 13:38:13.466740 32661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } }
I20260924 13:38:13.467209 754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b92a645f83a2488b9555d7c02550f35c (127.25.240.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } }
I20260924 13:38:13.467525 701 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:13.468082 754 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.240.125:46219
I20260924 13:38:13.468163 754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } }
I20260924 13:38:13.468233 754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:13.468292 658 master.cc:537] Master@127.25.240.125:46219 shutting down...
I20260924 13:38:13.468416 701 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Refusing update from remote peer 250ae62b280043d28245890fa6bbe456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 13:38:13.468858 755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:38:13.469627 32645 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 13:38:13.470219 32645 consensus_peers.cc:487] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 -> Peer b92a645f83a2488b9555d7c02550f35c (127.25.240.125:46219): Couldn't send request to peer b92a645f83a2488b9555d7c02550f35c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 13:38:13.471104 658 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:13.471190 658 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:13.471238 658 tablet_replica.cc:330] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: stopping tablet replica
I20260924 13:38:13.483665 658 master.cc:552] Master@127.25.240.125:46219 shutdown complete.
I20260924 13:38:13.483727 658 master_runner.cc:305] Clearing existing system tablet
I20260924 13:38:13.483848 658 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.483916 658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.484640 658 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:13.485157 759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.485272 658 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.485353 658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:13.485378 658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
uuid: "b92a645f83a2488b9555d7c02550f35c"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "54b471efd8fe825d866c957d0c532579"
server_key_iv: "64d05001aed02678ca9e46da4bb97aef"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.486030 658 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:13.487316 658 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 13:38:13.487370 658 log.cc:1196] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 13:38:13.487574 658 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Deleting consensus metadata
I20260924 13:38:13.487696 658 master_runner.cc:315] Copying system tablet from 127.25.240.126:35077
I20260924 13:38:13.488299 658 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.240.126:35077
I20260924 13:38:13.494709 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.488596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:38676)
0924 13:38:13.488691 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.488701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.488763 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.488905 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.488906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.489052 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.489334 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.489336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.490291 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.490292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.490816 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.490817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.491319 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.491320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.491365 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.491838 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.491843 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.492139 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.493811 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.493812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.493812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.493979 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.494183 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.494183 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.494184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.494233 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.494489 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.494490 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.494522 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.494571 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.494639 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":61,"threads_started":1}
I20260924 13:38:13.494881 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.488645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.25.240.126:35077)
0924 13:38:13.488691 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.488693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.488762 (+ 69us) server_negotiation.cc:407] Connection header received
0924 13:38:13.488822 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.488823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.488833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.488889 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.489407 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.490217 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.490894 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.491243 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.491316 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.492221 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.492223 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.492223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.492230 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.493739 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.494050 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.494051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.494052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.494112 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.494291 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.494292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.494292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.494419 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.494493 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.494593 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.494663 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:38:13.495365 32681 tablet_copy_service.cc:140] P 250ae62b280043d28245890fa6bbe456: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b92a645f83a2488b9555d7c02550f35c ({username='oryx', principal='oryx/127.25.240.125@KRBTEST.COM'} at 127.0.0.1:38676)
I20260924 13:38:13.495476 32681 tablet_copy_service.cc:161] P 250ae62b280043d28245890fa6bbe456: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b92a645f83a2488b9555d7c02550f35c at {username='oryx', principal='oryx/127.25.240.125@KRBTEST.COM'} at 127.0.0.1:38676: session id = b92a645f83a2488b9555d7c02550f35c-00000000000000000000000000000000
I20260924 13:38:13.496101 32681 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 13:38:13.496722 658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 13:38:13.497522 658 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: tablet copy: Starting download of 0 data blocks...
I20260924 13:38:13.497705 658 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: tablet copy: Starting download of 1 WAL segments...
I20260924 13:38:13.498538 658 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 13:38:13.499228 32681 tablet_copy_service.cc:342] P 250ae62b280043d28245890fa6bbe456: Request end of tablet copy session b92a645f83a2488b9555d7c02550f35c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.240.125@KRBTEST.COM'} at 127.0.0.1:38676
I20260924 13:38:13.499286 32681 tablet_copy_service.cc:433] P 250ae62b280043d28245890fa6bbe456: ending tablet copy session b92a645f83a2488b9555d7c02550f35c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b92a645f83a2488b9555d7c02550f35c
I20260924 13:38:13.501401 658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:13.502496 766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.502600 767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.502622 769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.502794 658 server_base.cc:1034] running on GCE node
I20260924 13:38:13.502882 658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:13.503021 658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:13.504158 658 hybrid_clock.cc:648] HybridClock initialized: now 1790257093504143 us; error 23 us; skew 500 ppm
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:13.505448 658 init.cc:377] Logged in from keytab as oryx/127.25.240.125@KRBTEST.COM (short username oryx)
I20260924 13:38:13.505716 658 webserver.cc:460] Webserver started at http://127.25.240.125:38155/ using document root <none> and password file <none>
I20260924 13:38:13.505844 658 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.505904 658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.506507 658 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:13.507012 776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.507131 658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.507174 658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
uuid: "b92a645f83a2488b9555d7c02550f35c"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "54b471efd8fe825d866c957d0c532579"
server_key_iv: "64d05001aed02678ca9e46da4bb97aef"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.507225 658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.527750 658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:13.529316 658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:13.532437 658 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.125:46219
I20260924 13:38:13.532598 828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.125:46219 every 8 connection(s)
I20260924 13:38:13.532852 658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
I20260924 13:38:13.533629 829 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:38:13.534058 829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Bootstrap starting.
I20260924 13:38:13.536962 829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:38:13.537184 829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: Bootstrap complete.
I20260924 13:38:13.537392 829 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } }
I20260924 13:38:13.537500 829 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b92a645f83a2488b9555d7c02550f35c, State: Initialized, Role: LEARNER
I20260924 13:38:13.537542 829 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } }
I20260924 13:38:13.537742 829 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:13.537741 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } }
I20260924 13:38:13.537997 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: This master's current role is: LEARNER
I20260924 13:38:13.539942 844 catalog_manager.cc:1259] Loaded cluster ID: 907e9bca243a4847b893d33ae34dd4b7
I20260924 13:38:13.540163 844 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: loading cluster ID for follower catalog manager: success
I20260924 13:38:13.541306 844 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: acquiring CA information for follower catalog manager: success
I20260924 13:38:13.541661 844 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:38:13.542618 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.534890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.25.240.125:46219)
0924 13:38:13.535048 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.535050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.535060 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:13.535084 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.535085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.535096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.535183 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.535831 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.536557 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.537384 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.537673 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.537741 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.539071 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.539073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.539073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.539080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.541417 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.541814 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.541815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.541816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.541870 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.542086 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.542087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.542087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.542196 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.542302 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.542416 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.542508 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.25.240.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.124.keytab.
Entry for principal oryx/127.25.240.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.124.keytab.
I20260924 13:38:13.568176 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.124:40639
--webserver_interface=127.25.240.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.124.keytab
--principal=oryx/127.25.240.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:35077,127.25.240.125:46219,127.25.240.124:40639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:13.648710 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:13.648904 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:13.648926 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:13.650138 847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:13.650179 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:13.650193 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:13.650235 847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:13.650260 847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:13.651685 847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:35077,127.25.240.125:46219,127.25.240.124:40639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.124.keytab
--principal=oryx/127.25.240.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.124:40639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.240.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:13.651921 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:13.652164 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:13.656950 853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.657035 847 server_base.cc:1034] running on GCE node
W20260924 13:38:13.656975 855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.656993 852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.657388 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:13.657752 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:13.658914 847 hybrid_clock.cc:648] HybridClock initialized: now 1790257093658908 us; error 34 us; skew 500 ppm
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:13.661212 847 init.cc:377] Logged in from keytab as oryx/127.25.240.124@KRBTEST.COM (short username oryx)
I20260924 13:38:13.661684 847 webserver.cc:460] Webserver started at http://127.25.240.124:37613/ using document root <none> and password file <none>
I20260924 13:38:13.661902 847 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.661966 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.662106 847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:13.663174 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data/instance:
uuid: "8491895e11604b73abdf8f675ebe8ef9"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "963a64aa41b8e0dec023a7be5b0d1c99"
server_key_iv: "b7c9a2833cbda1da61de295fb95f436a"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.663538 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal/instance:
uuid: "8491895e11604b73abdf8f675ebe8ef9"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "963a64aa41b8e0dec023a7be5b0d1c99"
server_key_iv: "b7c9a2833cbda1da61de295fb95f436a"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.664794 847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:13.665467 862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.665647 847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:13.665745 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
uuid: "8491895e11604b73abdf8f675ebe8ef9"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "963a64aa41b8e0dec023a7be5b0d1c99"
server_key_iv: "b7c9a2833cbda1da61de295fb95f436a"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.665824 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.682067 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:13.683833 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:13.683993 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:13.687988 847 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.124:40639
I20260924 13:38:13.688045 914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.124:40639 every 8 connection(s)
I20260924 13:38:13.688409 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
I20260924 13:38:13.689353 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:13.690860 915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:38:13.692932 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 847
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.124@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.240.124@KRBTEST.COM
I20260924 13:38:13.700714 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.691724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38690 (local address 127.25.240.126:35077)
0924 13:38:13.691835 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.691837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.691996 (+ 159us) server_negotiation.cc:407] Connection header received
0924 13:38:13.692545 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.692547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.692560 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.692605 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.693532 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.694063 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.695096 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.695395 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.695447 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.697832 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.697833 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.697833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.697841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.699240 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.699723 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.699725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.699725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.699788 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.700062 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.700063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.700063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.700215 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.700293 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.700535 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.700613 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 13:38:13.700783 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.691598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:38690)
0924 13:38:13.691864 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.691909 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.692432 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.692680 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.692684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.693108 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.693401 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.693407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.694171 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.694172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.695011 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.695014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.695486 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.695487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.695618 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.696141 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.696152 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.697745 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.699325 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.699330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.699332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.699653 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.699865 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.699865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.699866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.699993 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.700288 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.700293 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.700400 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.700485 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.700594 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":95,"threads_started":1}
I20260924 13:38:13.701395 915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:38:13.702571 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.693465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.25.240.124:40639)
0924 13:38:13.693611 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.693616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.693632 (+ 16us) server_negotiation.cc:407] Connection header received
0924 13:38:13.693651 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.693652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.693670 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.693756 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.694365 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.695123 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.696163 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.696446 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.696500 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.699119 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.699126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.699127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.699134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.701217 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.701613 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.701615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.701615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.701694 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.702013 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.702014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.702014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.702164 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.702305 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.702387 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.702465 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.124@KRBTEST.COM for oryx/127.25.240.125@KRBTEST.COM
I20260924 13:38:13.707464 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.701654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.0.0.1:56846)
0924 13:38:13.701703 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.701714 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.701746 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.701857 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.701857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.701956 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.702001 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.702002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.702579 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.702580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.703244 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.703246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.703575 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.703575 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.703612 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.703996 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.704000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.705093 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.706534 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.706535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.706535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.706714 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.706893 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.706894 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.706894 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.706943 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.707243 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.707243 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.707278 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.707331 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.707392 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:38:13.707526 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.701717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.25.240.125:46219)
0924 13:38:13.701759 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.701761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.701769 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:13.701793 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.701794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.701804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.701853 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.702059 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.702495 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.703314 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.703512 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.703566 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.705163 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.705164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.705164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.705170 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.706445 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.706781 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.706782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.706783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.706834 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.707050 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.707051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.707052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.707173 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.707227 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.707363 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.707425 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:13.707873 915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.124@KRBTEST.COM for oryx/127.25.240.124@KRBTEST.COM
I20260924 13:38:13.713430 917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.708144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:40639 (local address 127.0.0.1:49728)
0924 13:38:13.708237 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.708245 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.708284 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.708367 (+ 83us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.708368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.708475 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.708510 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.708511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.709047 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.709047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.709493 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.709494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.709858 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.709859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.709902 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.710198 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.710202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.711195 (+ 993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.712681 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.712681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.712681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.712833 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.712996 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.712996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.712996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.713042 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.713206 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.713206 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.713236 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.713284 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.713351 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 13:38:13.713478 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.708186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49728 (local address 127.25.240.124:40639)
0924 13:38:13.708229 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.708230 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.708273 (+ 43us) server_negotiation.cc:407] Connection header received
0924 13:38:13.708318 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.708319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.708326 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.708365 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.708548 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.708973 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.709597 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.709796 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.709847 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.711267 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.711268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.711269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.711274 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.712619 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.712892 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.712893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.712893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.712941 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.713079 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.713080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.713080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.713160 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.713202 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.713299 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.713393 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":29}
I20260924 13:38:13.714790 915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Bootstrap starting.
I20260924 13:38:13.715544 915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:13.715772 915 log.cc:824] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:13.716410 915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: No bootstrap required, opened a new log
I20260924 13:38:13.717921 915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } }
I20260924 13:38:13.718044 915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:13.718065 915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8491895e11604b73abdf8f675ebe8ef9, State: Initialized, Role: FOLLOWER
I20260924 13:38:13.718179 915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } }
I20260924 13:38:13.718415 920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } } }
I20260924 13:38:13.718479 915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:13.718526 920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:38:13.720220 931 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:38:13.720309 931 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:38:13.722554 847 master_runner.cc:428] Detected that this master 8491895e11604b73abdf8f675ebe8ef9 is joining an existing cluster
I20260924 13:38:13.722610 847 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.240.124:40639
I20260924 13:38:13.731632 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.724205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49738 (local address 127.25.240.124:40639)
0924 13:38:13.724239 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.724241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.724531 (+ 290us) server_negotiation.cc:407] Connection header received
0924 13:38:13.724611 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.724612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.724620 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.724646 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.725120 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.725860 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.726922 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.727189 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.727226 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.728425 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.728426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.728426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.728432 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.730572 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.730966 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.730967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.730968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.731017 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.731208 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.731208 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.731208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.731297 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.731345 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.731515 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.731577 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:13.731750 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.724198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:40639 (local address 127.0.0.1:49738)
0924 13:38:13.724485 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.724494 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.724549 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.724695 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.724696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.724984 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.725047 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.725048 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.725952 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.725952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.726828 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.726830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.727308 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.727309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.727367 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.727834 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.727840 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.728314 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.730662 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.730663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.730664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.730888 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.731077 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.731078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.731078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.731146 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.731359 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.731360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.731415 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.731500 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.731653 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":102,"thread_start_us":62,"threads_started":1}
I20260924 13:38:13.750144 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.724112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56852 (local address 127.25.240.125:46219)
0924 13:38:13.724172 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.724174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.724548 (+ 374us) server_negotiation.cc:407] Connection header received
0924 13:38:13.724570 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.724571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.724582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.724629 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.725118 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.725860 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.726922 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.727209 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.727242 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.747130 (+ 19888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.747132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.747132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.747141 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.748747 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.749188 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.749189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.749189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.749251 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.749475 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.749476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.749476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.749631 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.749676 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.749885 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.750051 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:38:13.750478 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.723881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.0.0.1:56852)
0924 13:38:13.724313 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.724321 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.724397 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.724670 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.724670 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.724957 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.725022 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.725024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.725952 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.725952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.726828 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.726830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.727309 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.727310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.727367 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.746476 (+ 19109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.746480 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.747033 (+ 553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.748888 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.748889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.748890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.749104 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.749331 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.749332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.749333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.749391 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.749724 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.749725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.749769 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.750250 (+ 481us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.750369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":8,"thread_start_us":72,"threads_started":1}
I20260924 13:38:13.751328 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.723941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38706 (local address 127.25.240.126:35077)
0924 13:38:13.723984 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.723986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.724413 (+ 427us) server_negotiation.cc:407] Connection header received
0924 13:38:13.724441 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.724442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.724455 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.724513 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.724965 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.725781 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.726905 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.727197 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.727245 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.746866 (+ 19621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.746868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.746869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.746877 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.749925 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.750479 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.750480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.750481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.750544 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.750763 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.750765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.750765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.750912 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.750985 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.751164 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.751243 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:38:13.751632 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.723882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:38706)
0924 13:38:13.724302 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.724312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.724376 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.724526 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.724527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.724795 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.724862 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.724864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.725877 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.725878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.726801 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.726804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.727330 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.727330 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.727389 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.746230 (+ 18841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.746237 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.746743 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.750018 (+ 3275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.750019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.750019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.750379 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.750615 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.750616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.750616 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.750679 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.751014 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.751014 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.751055 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.751420 (+ 365us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.751528 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260924 13:38:13.753872 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 32723
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257092, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.126@KRBTEST.COM for oryx/127.25.240.124@KRBTEST.COM
I20260924 13:38:13.761515 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:43813
--local_ip_for_outbound_sockets=127.25.240.65
--tserver_master_addrs=127.25.240.126:35077
--webserver_port=38753
--webserver_interface=127.25.240.65
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab
--principal=oryx/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219,127.25.240.124:40639 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:38:13.761584 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.753550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:40639 (local address 127.0.0.1:49748)
0924 13:38:13.753616 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.753628 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.753675 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.753862 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.753863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.754020 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.754064 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.754066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.754961 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.754962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.755869 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.755872 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.756337 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.756338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.756419 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.756841 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.756847 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.758517 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.760420 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.760421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.760421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.760660 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.760876 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.760876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.760877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.760945 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.761228 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.761229 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.761277 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.761355 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.761461 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 13:38:13.761647 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.753592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49748 (local address 127.25.240.124:40639)
0924 13:38:13.753626 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.753628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.753660 (+ 32us) server_negotiation.cc:407] Connection header received
0924 13:38:13.753753 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.753754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.753764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.753817 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.754149 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.754876 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.755956 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.756251 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.756320 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.758605 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.758606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.758606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.758613 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.760332 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.760744 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.760745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.760746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.760803 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.761020 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.761021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.761021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.761153 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.761216 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.761398 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.761547 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:13.762147 32661 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.240.124:40639: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9"
member_type: NON_VOTER
last_known_addr {
host: "127.25.240.124"
port: 40639
}
attrs {
promote: true
}
}
dest_uuid: "250ae62b280043d28245890fa6bbe456"
cas_config_opid_index: 5
I20260924 13:38:13.762382 32661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } }
I20260924 13:38:13.762857 754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8491895e11604b73abdf8f675ebe8ef9 (127.25.240.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.763898 754 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.240.124:40639
I20260924 13:38:13.763975 755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.764050 755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:13.764321 889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:13.765399 889 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Refusing update from remote peer 250ae62b280043d28245890fa6bbe456: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 13:38:13.765655 847 master.cc:537] Master@127.25.240.124:40639 shutting down...
I20260924 13:38:13.765657 755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:38:13.766322 32648 consensus_peers.cc:487] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 -> Peer 8491895e11604b73abdf8f675ebe8ef9 (127.25.240.124:40639): Couldn't send request to peer 8491895e11604b73abdf8f675ebe8ef9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 13:38:13.767918 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.762816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.0.0.1:56864)
0924 13:38:13.764261 (+ 1445us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.764272 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.764322 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.764527 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.764529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.764530 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:13.764565 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.764567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.765480 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.765481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.766738 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.766741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.767676 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.767678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.767753 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.767771 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.767822 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.767856 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1429}
I20260924 13:38:13.768239 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.764000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.25.240.125:46219)
0924 13:38:13.764061 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.764062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.764325 (+ 263us) server_negotiation.cc:407] Connection header received
0924 13:38:13.764384 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.764385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.764407 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.764475 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:13.764633 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.765385 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.766921 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.767496 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.767565 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.767633 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.768099 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.768138 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:38:13.768608 803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 LEARNER]: Refusing update from remote peer 250ae62b280043d28245890fa6bbe456: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 13:38:13.768836 755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 13:38:13.770265 803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8491895e11604b73abdf8f675ebe8ef9 (127.25.240.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.771472 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 250ae62b280043d28245890fa6bbe456. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.771564 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: This master's current role is: LEARNER
I20260924 13:38:13.771603 833 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: NON_VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: true } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.771657 833 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: This master's current role is: LEARNER
I20260924 13:38:13.771950 755 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: attempting to promote NON_VOTER b92a645f83a2488b9555d7c02550f35c to VOTER
I20260924 13:38:13.772138 755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } }
I20260924 13:38:13.772559 803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 LEARNER]: Refusing update from remote peer 250ae62b280043d28245890fa6bbe456: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260924 13:38:13.772981 32648 consensus_peers.cc:487] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 -> Peer 8491895e11604b73abdf8f675ebe8ef9 (127.25.240.124:40639): Couldn't send request to peer 8491895e11604b73abdf8f675ebe8ef9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 13:38:13.773083 754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 13:38:13.773252 847 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:13.773334 847 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:13.773375 847 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: stopping tablet replica
I20260924 13:38:13.774562 754 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b92a645f83a2488b9555d7c02550f35c (127.25.240.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.775518 754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.775606 754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:13.775668 754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.775725 754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:13.776006 803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b92a645f83a2488b9555d7c02550f35c (127.25.240.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.776960 942 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.777048 942 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:38:13.786227 847 master.cc:552] Master@127.25.240.124:40639 shutdown complete.
I20260924 13:38:13.786280 847 master_runner.cc:305] Clearing existing system tablet
I20260924 13:38:13.786407 847 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.786450 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.787474 847 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:38:13.788035 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.788182 847 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.788297 847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:38:13.788348 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
uuid: "8491895e11604b73abdf8f675ebe8ef9"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "963a64aa41b8e0dec023a7be5b0d1c99"
server_key_iv: "b7c9a2833cbda1da61de295fb95f436a"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.789048 847 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:13.790443 847 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 13:38:13.790535 847 log.cc:1196] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 13:38:13.790759 847 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Deleting consensus metadata
I20260924 13:38:13.790899 847 master_runner.cc:315] Copying system tablet from 127.25.240.126:35077
I20260924 13:38:13.791513 847 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: tablet copy: Beginning tablet copy session from remote peer at address 127.25.240.126:35077
I20260924 13:38:13.798714 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.791825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:38718)
0924 13:38:13.791942 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.791953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.792008 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.792154 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.792155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.792375 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.792423 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.792425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.793362 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.793362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.794174 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.794176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.794627 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.794628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.794682 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.795212 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.795217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.795595 (+ 378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.797495 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.797496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.797496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.797760 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.798063 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.798064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.798065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.798132 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.798398 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.798399 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.798448 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.798520 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.798618 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260924 13:38:13.798731 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.791893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.25.240.126:35077)
0924 13:38:13.791934 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.791936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.791997 (+ 61us) server_negotiation.cc:407] Connection header received
0924 13:38:13.792061 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.792061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.792071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.792148 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.792510 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.793271 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.794272 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.794544 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.794621 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.795668 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.795669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.795669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.795677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.797419 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.797924 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.797926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.797926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.797990 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.798198 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.798198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.798199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.798327 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.798395 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.798560 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.798634 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:38:13.799346 32681 tablet_copy_service.cc:140] P 250ae62b280043d28245890fa6bbe456: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8491895e11604b73abdf8f675ebe8ef9 ({username='oryx', principal='oryx/127.25.240.124@KRBTEST.COM'} at 127.0.0.1:38718)
I20260924 13:38:13.799413 32681 tablet_copy_service.cc:161] P 250ae62b280043d28245890fa6bbe456: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8491895e11604b73abdf8f675ebe8ef9 at {username='oryx', principal='oryx/127.25.240.124@KRBTEST.COM'} at 127.0.0.1:38718: session id = 8491895e11604b73abdf8f675ebe8ef9-00000000000000000000000000000000
I20260924 13:38:13.799974 32681 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 13:38:13.800365 847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 13:38:13.801617 847 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: tablet copy: Starting download of 0 data blocks...
I20260924 13:38:13.801810 847 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: tablet copy: Starting download of 1 WAL segments...
I20260924 13:38:13.802594 847 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 13:38:13.803450 32681 tablet_copy_service.cc:342] P 250ae62b280043d28245890fa6bbe456: Request end of tablet copy session 8491895e11604b73abdf8f675ebe8ef9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.240.124@KRBTEST.COM'} at 127.0.0.1:38718
I20260924 13:38:13.803501 32681 tablet_copy_service.cc:433] P 250ae62b280043d28245890fa6bbe456: ending tablet copy session 8491895e11604b73abdf8f675ebe8ef9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8491895e11604b73abdf8f675ebe8ef9
I20260924 13:38:13.806550 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:13.807829 952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.807868 955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.808429 847 server_base.cc:1034] running on GCE node
W20260924 13:38:13.810348 953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.810494 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:13.810627 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:13.811770 847 hybrid_clock.cc:648] HybridClock initialized: now 1790257093811759 us; error 57 us; skew 500 ppm
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:13.813272 847 init.cc:377] Logged in from keytab as oryx/127.25.240.124@KRBTEST.COM (short username oryx)
I20260924 13:38:13.813522 847 webserver.cc:460] Webserver started at http://127.25.240.124:42393/ using document root <none> and password file <none>
I20260924 13:38:13.813657 847 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.813696 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.814452 847 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:38:13.815025 962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.815176 847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:13.815227 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
uuid: "8491895e11604b73abdf8f675ebe8ef9"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "963a64aa41b8e0dec023a7be5b0d1c99"
server_key_iv: "b7c9a2833cbda1da61de295fb95f436a"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.815277 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.820367 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:13.822271 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:13.825294 847 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.124:40639
I20260924 13:38:13.825335 1014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.124:40639 every 8 connection(s)
I20260924 13:38:13.825825 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
I20260924 13:38:13.826754 1015 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:38:13.827252 1015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Bootstrap starting.
I20260924 13:38:13.830016 1015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:38:13.830250 1015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Bootstrap complete.
I20260924 13:38:13.830420 1015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } }
I20260924 13:38:13.830502 1015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8491895e11604b73abdf8f675ebe8ef9, State: Initialized, Role: LEARNER
I20260924 13:38:13.830533 1015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } }
I20260924 13:38:13.830737 1015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:13.830750 1018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:13.830961 1018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: This master's current role is: LEARNER
I20260924 13:38:13.832487 1029 catalog_manager.cc:1259] Loaded cluster ID: 907e9bca243a4847b893d33ae34dd4b7
I20260924 13:38:13.832566 1029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: loading cluster ID for follower catalog manager: success
I20260924 13:38:13.833865 1029 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: acquiring CA information for follower catalog manager: success
I20260924 13:38:13.834216 1029 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 13:38:13.859475 940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:13.859661 940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:13.859684 940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:13.860790 940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:13.860824 940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:13.860850 940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:13.862324 940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab
--principal=oryx/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:43813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=38753
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219,127.25.240.124:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:13.862520 940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:13.862754 940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:13.868763 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.868824 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:13.868973 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:13.869038 940 server_base.cc:1034] running on GCE node
I20260924 13:38:13.869350 940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:13.869640 940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:13.870807 940 hybrid_clock.cc:648] HybridClock initialized: now 1790257093870789 us; error 35 us; skew 500 ppm
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:13.873327 940 init.cc:377] Logged in from keytab as oryx/127.25.240.65@KRBTEST.COM (short username oryx)
I20260924 13:38:13.873801 940 webserver.cc:460] Webserver started at http://127.25.240.65:38753/ using document root <none> and password file <none>
I20260924 13:38:13.874037 940 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:13.874115 940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:13.875483 940 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:13.876091 1044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.876250 940 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:13.876344 940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "d2135c02dfb042069cfb5a9747cbff71"
format_stamp: "Formatted at 2026-09-24 13:38:12 on dist-test-slave-m5z1"
server_key: "860fa0bd286c5fe213262136344091ca"
server_key_iv: "7adb49493adfd3a53fcdcc58a7ae678e"
server_key_version: "encryptionkey@0"
I20260924 13:38:13.876685 940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:13.901899 940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:13.903692 940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:13.903887 940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:13.904130 940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:13.904438 940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:13.904515 940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.904553 940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:13.904579 940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:13.910759 940 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:43813
I20260924 13:38:13.910822 1157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:43813 every 8 connection(s)
I20260924 13:38:13.911158 940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.124@KRBTEST.COM
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.125@KRBTEST.COM
Sep 24 13:38:13 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257093, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:13.919520 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 940
I20260924 13:38:13.919677 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 389
I20260924 13:38:13.922731 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.911907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:45261 (local address 127.25.240.126:35077)
0924 13:38:13.911953 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.911955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.912213 (+ 258us) server_negotiation.cc:407] Connection header received
0924 13:38:13.912591 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.912592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.912605 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.912636 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.913591 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.914336 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.915545 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.915833 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.915897 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.919587 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.919589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.919589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.919598 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.921451 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.921901 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.921902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.921903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.921959 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.922154 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.922155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.922155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.922287 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.922330 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.922540 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.922627 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:38:13.923309 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.911834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:39019 (local address 127.25.240.124:40639)
0924 13:38:13.912275 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.912278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.912285 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:13.912643 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.912644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.912656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.912731 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.913581 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.914233 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.915633 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.915992 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.916074 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.918517 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.918518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.918519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.918525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.920938 (+ 2413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.921360 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.921362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.921362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.921426 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.921759 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.921760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.921761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.921906 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.921977 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.923071 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.923166 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":56,"threads_started":1}
I20260924 13:38:13.922734 1165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.911814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:40639 (local address 127.25.240.65:39019)
0924 13:38:13.912088 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.912099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.912495 (+ 396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.913192 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.913193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.913390 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.913449 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.913456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.914339 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.914340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.915552 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.915554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.916133 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.916134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.916184 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.916808 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.916813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.918430 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.921038 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.921045 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.921047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.921269 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.921503 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.921504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.921504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.921683 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.922003 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.922010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.922171 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.922276 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.922455 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":45,"thread_start_us":73,"threads_started":1}
I20260924 13:38:13.923002 1163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.911814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.65:45261)
0924 13:38:13.912162 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.912171 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.912495 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.912839 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.912845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.913291 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.913449 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.913456 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.914413 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.914414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.915455 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.915458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.915918 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.915919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.916069 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.916663 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.916671 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.919463 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.921541 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.921542 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.921543 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.921815 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.922029 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.922030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.922030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.922088 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.922373 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.922373 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.922417 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.922837 (+ 420us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.922936 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":198,"thread_start_us":63,"threads_started":1}
I20260924 13:38:13.923902 1159 heartbeater.cc:347] Connected to a master server at 127.25.240.124:40639
I20260924 13:38:13.924008 1159 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:13.924189 1159 heartbeater.cc:511] Master 127.25.240.124:40639 requested a full tablet report, sending...
I20260924 13:38:13.924705 978 ts_manager.cc:194] Registered new tserver with Master: d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813)
I20260924 13:38:13.926144 1158 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35077
I20260924 13:38:13.926203 1158 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:13.926282 1158 heartbeater.cc:511] Master 127.25.240.126:35077 requested a full tablet report, sending...
I20260924 13:38:13.926340 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:39671
--local_ip_for_outbound_sockets=127.25.240.66
--tserver_master_addrs=127.25.240.126:35077
--webserver_port=33629
--webserver_interface=127.25.240.66
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab
--principal=oryx/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219,127.25.240.124:40639 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:38:13.926566 32661 ts_manager.cc:194] Re-registered known tserver with Master: d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813)
I20260924 13:38:13.927065 32661 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:45261
I20260924 13:38:13.928478 1164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.911814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.25.240.65:36787)
0924 13:38:13.911980 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:13.912021 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:13.912476 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:13.912841 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:13.912844 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:13.913266 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:13.913449 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.913453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.914640 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.914640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.915553 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:13.915554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.915990 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.915991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:13.916069 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.916661 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:13.916671 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:13.918913 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:13.927370 (+ 8457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.927371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.927372 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.927613 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.927861 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:13.927861 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:13.927862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:13.927913 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:13.928231 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:13.928232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:13.928264 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 13:38:13.928320 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:38:13.928400 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":24,"thread_start_us":89,"threads_started":1}
I20260924 13:38:13.928547 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:13.912036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:36787 (local address 127.25.240.125:46219)
0924 13:38:13.912078 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:13.912080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:13.912089 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:13.912738 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:13.912740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:13.912771 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:13.912829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:13.913580 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.914343 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.915656 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:13.915856 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:13.915915 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:13.919001 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:13.919003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:13.919003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:13.919011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:13.927182 (+ 8171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.927714 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.927716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.927716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.927786 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:13.927988 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:13.927989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:13.927989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:13.928154 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:13.928244 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:13.928354 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 13:38:13.928441 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:38:13.928846 1160 heartbeater.cc:347] Connected to a master server at 127.25.240.125:46219
I20260924 13:38:13.928895 1160 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:13.928992 1160 heartbeater.cc:511] Master 127.25.240.125:46219 requested a full tablet report, sending...
I20260924 13:38:13.929333 793 ts_manager.cc:194] Registered new tserver with Master: d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813)
W20260924 13:38:14.008554 1166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:14.008774 1166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:14.008810 1166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:14.009956 1166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:14.010021 1166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:14.010066 1166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:14.011528 1166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab
--principal=oryx/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:39671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=33629
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219,127.25.240.124:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:14.011791 1166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:14.012017 1166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:14.016925 1171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:14.016964 1172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:14.017210 1166 server_base.cc:1034] running on GCE node
W20260924 13:38:14.016942 1174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:14.017467 1166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:14.017724 1166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:14.018867 1166 hybrid_clock.cc:648] HybridClock initialized: now 1790257094018847 us; error 30 us; skew 500 ppm
Sep 24 13:38:14 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257094, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:14.021234 1166 init.cc:377] Logged in from keytab as oryx/127.25.240.66@KRBTEST.COM (short username oryx)
I20260924 13:38:14.021741 1166 webserver.cc:460] Webserver started at http://127.25.240.66:33629/ using document root <none> and password file <none>
I20260924 13:38:14.021982 1166 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:14.022063 1166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:14.023324 1166 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:14.023896 1181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:14.024065 1166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:14.024160 1166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "0dd242ac904746a9a95f1fda67689068"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "5a64223ac4c6b5213efbdb259cfd5954"
server_key_iv: "a2da9317b41d857a795a11d2b1cb6966"
server_key_version: "encryptionkey@0"
I20260924 13:38:14.024483 1166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:14.042739 1166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:14.044461 1166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:14.044612 1166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:14.044836 1166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:14.045130 1166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:14.045182 1166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:14.045223 1166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:14.045250 1166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:14.051139 1166 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:39671
I20260924 13:38:14.051175 1294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:39671 every 8 connection(s)
I20260924 13:38:14.051589 1166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:38:14 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257094, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.124@KRBTEST.COM
Sep 24 13:38:14 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257094, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
Sep 24 13:38:14 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257094, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.125@KRBTEST.COM
I20260924 13:38:14.061890 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 1166
I20260924 13:38:14.062076 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 524
I20260924 13:38:14.063783 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.052340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:53891 (local address 127.25.240.125:46219)
0924 13:38:14.052410 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.052412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.052658 (+ 246us) server_negotiation.cc:407] Connection header received
0924 13:38:14.053807 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.053809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.053821 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.053875 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.055982 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.056680 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.057606 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.057850 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.057918 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.060657 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.060658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.060658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.060665 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.062354 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.062818 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.062820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.062820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.062875 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.063135 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.063136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.063136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.063253 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.063330 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.063621 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.063696 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:38:14.063977 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.052313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.25.240.66:53891)
0924 13:38:14.052571 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.052581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.053539 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.053901 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.053902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.054246 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:14.054411 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.054417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.056752 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.056753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.057495 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.057498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.057982 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.057983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.058059 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.058483 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:14.058488 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:14.060590 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:14.062436 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.062437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.062438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.062725 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.062954 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.062955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.062956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.063067 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.063321 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:14.063327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:14.063488 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.063590 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.063737 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":26,"thread_start_us":97,"threads_started":1}
I20260924 13:38:14.064522 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.052316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:42041 (local address 127.25.240.126:35077)
0924 13:38:14.052452 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.052454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.052652 (+ 198us) server_negotiation.cc:407] Connection header received
0924 13:38:14.053688 (+ 1036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.053689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.053702 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.053738 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.054512 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.055020 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.056458 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.056769 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.056806 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.060170 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.060172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.060172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.060180 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.062022 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.063397 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.063398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.063398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.063457 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.063680 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.063682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.063682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.063822 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.063866 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.064331 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.064425 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260924 13:38:14.064548 1297 heartbeater.cc:347] Connected to a master server at 127.25.240.125:46219
I20260924 13:38:14.064561 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.052480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.66:42041)
0924 13:38:14.052593 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.052603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.053540 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.053846 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.053849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.054208 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:14.054410 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.054414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.055129 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.055131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.056351 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.056353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.056900 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.056901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.057044 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.057851 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:14.057856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:14.060050 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:14.062096 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.062102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.062105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.063296 (+ 1191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.063528 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.063529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.063530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.063599 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.063963 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:14.063964 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:14.064195 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.064280 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.064391 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":5,"thread_start_us":67,"threads_started":1}
I20260924 13:38:14.064632 1297 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:14.064838 1297 heartbeater.cc:511] Master 127.25.240.125:46219 requested a full tablet report, sending...
I20260924 13:38:14.064867 1295 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35077
I20260924 13:38:14.064939 1295 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:14.065022 1295 heartbeater.cc:511] Master 127.25.240.126:35077 requested a full tablet report, sending...
I20260924 13:38:14.065292 793 ts_manager.cc:194] Registered new tserver with Master: 0dd242ac904746a9a95f1fda67689068 (127.25.240.66:39671)
I20260924 13:38:14.065306 32661 ts_manager.cc:194] Re-registered known tserver with Master: 0dd242ac904746a9a95f1fda67689068 (127.25.240.66:39671)
I20260924 13:38:14.065763 32661 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:42041
I20260924 13:38:14.067687 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.052397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:45689 (local address 127.25.240.124:40639)
0924 13:38:14.052451 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.052453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.052642 (+ 189us) server_negotiation.cc:407] Connection header received
0924 13:38:14.053687 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.053688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.053703 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.053760 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.055142 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.055866 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.056559 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.056813 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.056882 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.059688 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.059690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.059690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.059696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.061664 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.066953 (+ 5289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.066953 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.066954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.067007 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.067215 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.067216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.067216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.067310 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.067356 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.067538 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.067624 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:38:14.067839 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.052313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:40639 (local address 127.25.240.66:45689)
0924 13:38:14.052514 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.052555 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.053544 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.053844 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.053848 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.054242 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:14.054411 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.054417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.055947 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.055947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.056467 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.056468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.056900 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.056901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.057044 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.057777 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:14.057788 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:14.059604 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:14.066152 (+ 6548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.066153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.066154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.066863 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.067075 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.067075 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.067076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.067147 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.067371 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:14.067371 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:14.067415 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.067482 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.067603 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":35,"thread_start_us":85,"threads_started":1}
I20260924 13:38:14.068166 1296 heartbeater.cc:347] Connected to a master server at 127.25.240.124:40639
I20260924 13:38:14.068217 1296 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:14.068289 1296 heartbeater.cc:511] Master 127.25.240.124:40639 requested a full tablet report, sending...
I20260924 13:38:14.068617 978 ts_manager.cc:194] Registered new tserver with Master: 0dd242ac904746a9a95f1fda67689068 (127.25.240.66:39671)
I20260924 13:38:14.068733 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:36131
--local_ip_for_outbound_sockets=127.25.240.67
--tserver_master_addrs=127.25.240.126:35077
--webserver_port=33711
--webserver_interface=127.25.240.67
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab
--principal=oryx/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219,127.25.240.124:40639 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:14.149724 1302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:14.149962 1302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:14.150002 1302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:14.151134 1302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:14.151199 1302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:14.151244 1302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:14.152724 1302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab
--principal=oryx/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:36131
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=33711
--tserver_master_addrs=127.25.240.126:35077,127.25.240.125:46219,127.25.240.124:40639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:14.152993 1302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:14.153218 1302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:14.158025 1307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:14.158020 1308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:14.158092 1310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:14.158164 1302 server_base.cc:1034] running on GCE node
I20260924 13:38:14.158551 1302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:14.158811 1302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:14.159968 1302 hybrid_clock.cc:648] HybridClock initialized: now 1790257094159963 us; error 36 us; skew 500 ppm
Sep 24 13:38:14 dist-test-slave-m5z1 krb5kdc[32616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257094, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:14.162454 1302 init.cc:377] Logged in from keytab as oryx/127.25.240.67@KRBTEST.COM (short username oryx)
I20260924 13:38:14.162865 1302 webserver.cc:460] Webserver started at http://127.25.240.67:33711/ using document root <none> and password file <none>
I20260924 13:38:14.163055 1302 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:14.163106 1302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:14.164341 1302 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:38:14.164877 1317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:14.165081 1302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:14.165149 1302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "45fda73093f549059ff1d2f8ef9e411e"
format_stamp: "Formatted at 2026-09-24 13:38:13 on dist-test-slave-m5z1"
server_key: "56b8b95c4464ba002ff42fd2badf94bb"
server_key_iv: "7399842ec2d842b5165530f6bbd68845"
server_key_version: "encryptionkey@0"
I20260924 13:38:14.165491 1302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:14.173187 1302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:14.174890 1302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:14.175033 1302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:14.175241 1302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:14.175529 1302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:14.175583 1302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:14.175624 1302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:14.175652 1302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:14.181368 1302 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:36131
I20260924 13:38:14.181411 1430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:36131 every 8 connection(s)
I20260924 13:38:14.181788 1302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:38:14.182925 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 1302
Sep 24 13:38:14 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257094, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.125@KRBTEST.COM
Sep 24 13:38:14 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257094, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.124@KRBTEST.COM
I20260924 13:38:14.192690 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.184145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49762 (local address 127.25.240.124:40639)
0924 13:38:14.184927 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.184930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.185042 (+ 112us) server_negotiation.cc:407] Connection header received
0924 13:38:14.185063 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.185064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.185075 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.185131 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.185966 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.186660 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.187645 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.187949 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.188007 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.189480 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.189482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.189482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.189489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.191280 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.191649 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.191651 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.191651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.191708 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.191906 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.191907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.191908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.192053 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.192143 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.192482 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.192564 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":47,"threads_started":1}
I20260924 13:38:14.194361 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.184533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38734 (local address 127.25.240.126:35077)
0924 13:38:14.185123 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.185125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.185222 (+ 97us) server_negotiation.cc:407] Connection header received
0924 13:38:14.185316 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.185317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.185329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.185376 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.186144 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.186885 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.187954 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.188265 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.188304 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.190507 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.190508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.190509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.190515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.192703 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.193429 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.193430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.193431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.193494 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.193754 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.193755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.193755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.193884 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.193972 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.194184 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.194266 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":77,"threads_started":1}
Sep 24 13:38:14 dist-test-slave-m5z1 krb5kdc[32616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257094, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:14.196326 831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.182737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:56109 (local address 127.25.240.125:46219)
0924 13:38:14.182938 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.182940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.182955 (+ 15us) server_negotiation.cc:407] Connection header received
0924 13:38:14.183423 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.183424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.183436 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.183515 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.184348 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.185078 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.186736 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.187025 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.187086 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.190558 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.190559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.190560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.190567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.192843 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.193312 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.193313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.193313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.193360 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.193964 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.193964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.193965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.194090 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.194150 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.195223 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.195566 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260924 13:38:14.197242 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.182419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.25.240.67:56109)
0924 13:38:14.182850 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.182859 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.183328 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.183861 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.183862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.184075 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:14.184258 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.184264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.185193 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.185194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.186637 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.186639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.187143 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.187143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.187274 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.188001 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:14.188005 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:14.190443 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:14.192927 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.192933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.192935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.193240 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.193760 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.193761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.193761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.193885 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.194176 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:14.194182 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:14.194329 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.194686 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.197029 (+ 2343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":193,"thread_start_us":67,"threads_started":1}
I20260924 13:38:14.198206 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.185178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.25.240.125:46219)
0924 13:38:14.188639 (+ 3461us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.188641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.188648 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:14.188667 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.188668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.188680 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.190533 (+ 1853us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.190569 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.191331 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.192403 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.193121 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.193213 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.194552 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.194553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.194554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.194560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.196640 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.197054 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.197055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.197055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.197109 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.197313 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.197314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.197314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.197437 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.197984 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.198025 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.198099 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3441,"thread_start_us":58,"threads_started":1}
I20260924 13:38:14.198987 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.182672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:58711 (local address 127.25.240.124:40639)
0924 13:38:14.182730 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.182733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.182741 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:14.183454 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.183456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.183469 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.183501 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.184366 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.185094 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.186730 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.187029 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.187067 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.195760 (+ 8693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.195761 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.195762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.195769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.197432 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.198255 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.198256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.198257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.198309 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.198501 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.198502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.198503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.198614 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.198688 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.198823 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.198894 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 13:38:14.196781 32661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 13:38:14.198995 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.182418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:40639 (local address 127.25.240.67:58711)
0924 13:38:14.182587 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.182629 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.183320 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.183587 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.183593 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.183980 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:14.184257 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.184265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.185298 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.185298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.186638 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.186641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.187148 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.187148 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.187274 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.187908 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:14.187918 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:14.195107 (+ 7189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:14.197971 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.197972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.197972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.198167 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.198382 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.198382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.198382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.198426 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.198681 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:14.198681 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:14.198720 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.198781 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.198881 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":58,"thread_start_us":67,"threads_started":1}
W20260924 13:38:14.199199 32661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:14.199362 1432 heartbeater.cc:347] Connected to a master server at 127.25.240.124:40639
I20260924 13:38:14.199491 1432 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:14.199695 1432 heartbeater.cc:511] Master 127.25.240.124:40639 requested a full tablet report, sending...
I20260924 13:38:14.200233 1433 heartbeater.cc:347] Connected to a master server at 127.25.240.125:46219
I20260924 13:38:14.200249 978 ts_manager.cc:194] Registered new tserver with Master: 45fda73093f549059ff1d2f8ef9e411e (127.25.240.67:36131)
I20260924 13:38:14.200286 1433 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:14.200345 1433 heartbeater.cc:511] Master 127.25.240.125:46219 requested a full tablet report, sending...
I20260924 13:38:14.200909 792 ts_manager.cc:194] Registered new tserver with Master: 45fda73093f549059ff1d2f8ef9e411e (127.25.240.67:36131)
I20260924 13:38:14.201725 32700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.182469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:59323 (local address 127.25.240.126:35077)
0924 13:38:14.182516 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.182518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.182744 (+ 226us) server_negotiation.cc:407] Connection header received
0924 13:38:14.183994 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.183995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.184009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.184071 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.190832 (+ 6761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.191585 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.194244 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.194574 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.194613 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.197724 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.197726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.197726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.197733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.199987 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.200862 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.200863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.200863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.200925 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.201136 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.201137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.201138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.201276 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.201354 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.201531 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.201643 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:38:14.201915 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.182418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.67:59323)
0924 13:38:14.182666 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.182676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.183882 (+ 1206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.184119 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.184120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.184268 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:14.185546 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.185549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.192020 (+ 6471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.192021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.194132 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.194135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.194684 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.194684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.194753 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.195216 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:14.195221 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:14.197636 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:14.200065 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.200066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.200067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.200267 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.201005 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:14.201006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:14.201007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:14.201067 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:14.201359 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:14.201360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:14.201402 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.201467 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.201608 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260924 13:38:14.202354 1431 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35077
I20260924 13:38:14.202410 1431 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:14.202466 1431 heartbeater.cc:511] Master 127.25.240.126:35077 requested a full tablet report, sending...
I20260924 13:38:14.202664 32660 ts_manager.cc:194] Re-registered known tserver with Master: 45fda73093f549059ff1d2f8ef9e411e (127.25.240.67:36131)
I20260924 13:38:14.203106 32660 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:59323
I20260924 13:38:14.207449 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.203651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:39671 (local address 127.0.0.1:47240)
0924 13:38:14.203692 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.203703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.203737 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.204159 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.204160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.204161 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:14.204213 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.204215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.205228 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.205229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.206332 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.206335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.207234 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.207235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.207293 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.207296 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.207339 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.207379 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 13:38:14.207684 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.203818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.25.240.66:39671)
0924 13:38:14.203995 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.204000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.204010 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:14.204052 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.204054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.204070 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.204167 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:14.204271 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.205145 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.206424 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.207147 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.207207 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.207544 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.207573 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.207610 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
I20260924 13:38:14.207808 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.203715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:36131 (local address 127.0.0.1:33332)
0924 13:38:14.203837 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.203847 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.203899 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.204448 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.204449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.204451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:14.204493 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.204494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.205759 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.205761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.206691 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.206693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.207600 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.207602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.207671 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.207675 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.207702 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.207736 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":80,"threads_started":1}
I20260924 13:38:14.208118 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.204111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.25.240.67:36131)
0924 13:38:14.204262 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.204267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.204275 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:14.204306 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.204307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.204320 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.204561 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:14.204606 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.205661 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.206822 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.207500 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.207918 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.207995 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.208018 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.208052 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260924 13:38:14.208138 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.203882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:43813 (local address 127.0.0.1:44858)
0924 13:38:14.203995 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.204004 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.204044 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.204614 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.204615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.204618 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:14.204678 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.204680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.205869 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.205870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.206979 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.206982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.207963 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.207964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.208038 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.208041 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.208066 (+ 25us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.208094 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260924 13:38:14.208362 1229 tablet_service.cc:1467] Processing CreateTablet for tablet 55706974a05a4c39924418472c4b075a (DEFAULT_TABLE table=test-table [id=cb9077d5f38243e19fe17825d7d4d428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:14.208461 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.204256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44858 (local address 127.25.240.65:43813)
0924 13:38:14.204446 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.204450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.204459 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:14.204494 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.204496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.204512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.204612 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:14.204745 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.205791 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.207074 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.207833 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.208247 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.208325 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.208349 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.208388 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
I20260924 13:38:14.208702 1229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55706974a05a4c39924418472c4b075a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:14.208712 1365 tablet_service.cc:1467] Processing CreateTablet for tablet 55706974a05a4c39924418472c4b075a (DEFAULT_TABLE table=test-table [id=cb9077d5f38243e19fe17825d7d4d428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:14.209008 1365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55706974a05a4c39924418472c4b075a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:14.210319 1092 tablet_service.cc:1467] Processing CreateTablet for tablet 55706974a05a4c39924418472c4b075a (DEFAULT_TABLE table=test-table [id=cb9077d5f38243e19fe17825d7d4d428]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:14.210628 1092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55706974a05a4c39924418472c4b075a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:14.211653 1457 tablet_bootstrap.cc:492] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: Bootstrap starting.
I20260924 13:38:14.212201 1457 tablet_bootstrap.cc:654] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:14.212412 1457 log.cc:824] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:14.212833 1458 tablet_bootstrap.cc:492] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Bootstrap starting.
I20260924 13:38:14.213435 1458 tablet_bootstrap.cc:654] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:14.213768 1458 log.cc:824] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:14.214073 1457 tablet_bootstrap.cc:492] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: No bootstrap required, opened a new log
I20260924 13:38:14.214152 1457 ts_tablet_manager.cc:1397] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:38:14.214169 1461 tablet_bootstrap.cc:492] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: Bootstrap starting.
I20260924 13:38:14.214836 1458 tablet_bootstrap.cc:492] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: No bootstrap required, opened a new log
I20260924 13:38:14.214887 1458 ts_tablet_manager.cc:1397] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:14.216183 1457 raft_consensus.cc:348] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.216180 1458 raft_consensus.cc:348] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.216336 1458 raft_consensus.cc:374] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:14.216337 1457 raft_consensus.cc:374] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:14.216377 1458 raft_consensus.cc:729] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd242ac904746a9a95f1fda67689068, State: Initialized, Role: FOLLOWER
I20260924 13:38:14.216377 1457 raft_consensus.cc:729] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fda73093f549059ff1d2f8ef9e411e, State: Initialized, Role: FOLLOWER
I20260924 13:38:14.216485 1458 consensus_queue.cc:260] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.216476 1457 consensus_queue.cc:260] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.216718 1458 ts_tablet_manager.cc:1428] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:14.216737 1457 ts_tablet_manager.cc:1428] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:38:14.216737 1295 heartbeater.cc:503] Master 127.25.240.126:35077 was elected leader, sending a full tablet report...
I20260924 13:38:14.216836 1431 heartbeater.cc:503] Master 127.25.240.126:35077 was elected leader, sending a full tablet report...
I20260924 13:38:14.217048 1461 tablet_bootstrap.cc:654] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:14.217330 1461 log.cc:824] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:14.218410 1461 tablet_bootstrap.cc:492] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: No bootstrap required, opened a new log
I20260924 13:38:14.218474 1461 ts_tablet_manager.cc:1397] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 13:38:14.219744 1461 raft_consensus.cc:348] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.219858 1461 raft_consensus.cc:374] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:14.219879 1461 raft_consensus.cc:729] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2135c02dfb042069cfb5a9747cbff71, State: Initialized, Role: FOLLOWER
I20260924 13:38:14.219982 1461 consensus_queue.cc:260] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.220216 1461 ts_tablet_manager.cc:1428] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:14.220232 1158 heartbeater.cc:503] Master 127.25.240.126:35077 was elected leader, sending a full tablet report...
I20260924 13:38:14.225268 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.222925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:40639 (local address 127.0.0.1:49764)
0924 13:38:14.222980 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.222991 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.223031 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.223179 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.223180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.223181 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:14.223223 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.223224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.223843 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.223844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.224556 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.224558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.225106 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.225106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.225149 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.225152 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.225191 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.225222 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:38:14.225279 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.222993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49764 (local address 127.25.240.124:40639)
0924 13:38:14.223029 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.223031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.223039 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:14.223095 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.223096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.223115 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.223177 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:14.223270 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.223755 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.224644 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.225036 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.225096 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.225136 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.225207 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.225234 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:38:14.225633 989 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 13:38:14.226470 1018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 250ae62b280043d28245890fa6bbe456. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:14.226557 1018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: This master's current role is: LEARNER
W20260924 13:38:14.302317 1298 tablet.cc:2369] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:38:14.333989 1463 raft_consensus.cc:482] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:14.334101 1463 raft_consensus.cc:504] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.334489 1463 leader_election.cc:288] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dd242ac904746a9a95f1fda67689068 (127.25.240.66:39671), d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813)
I20260924 13:38:14.338368 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.334749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:39893 (local address 127.25.240.65:43813)
0924 13:38:14.334809 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.334815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.334824 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:14.334849 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.334850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.334858 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.334936 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:14.335149 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.335905 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.337423 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.338010 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.338075 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.338085 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.338248 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.338302 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:38:14.338368 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.334852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:49615 (local address 127.25.240.66:39671)
0924 13:38:14.334921 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.334922 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.334931 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:14.334954 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.334956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.334964 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.335040 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:14.335157 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.335904 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.337423 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.337996 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.338075 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.338085 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.338248 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.338284 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:38:14.338359 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.334623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:39671 (local address 127.25.240.67:49615)
0924 13:38:14.334672 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.334683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.334732 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.335049 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.335050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.335051 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:14.335091 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.335092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.335983 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.335983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.337332 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.337335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.338079 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.338080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.338164 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.338168 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.338229 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.338271 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":97}
I20260924 13:38:14.338367 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.334675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:43813 (local address 127.25.240.67:39893)
0924 13:38:14.334716 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:14.334725 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:14.334757 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:14.334965 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:14.334966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:14.334967 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:14.335069 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.335072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.335983 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.335983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.337332 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:14.337335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.338079 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.338080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:14.338164 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.338168 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:14.338229 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:38:14.338283 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":74}
I20260924 13:38:14.338946 1112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55706974a05a4c39924418472c4b075a" candidate_uuid: "45fda73093f549059ff1d2f8ef9e411e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2135c02dfb042069cfb5a9747cbff71" is_pre_election: true
I20260924 13:38:14.339120 1112 raft_consensus.cc:2454] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45fda73093f549059ff1d2f8ef9e411e in term 0.
I20260924 13:38:14.339257 1249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55706974a05a4c39924418472c4b075a" candidate_uuid: "45fda73093f549059ff1d2f8ef9e411e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd242ac904746a9a95f1fda67689068" is_pre_election: true
I20260924 13:38:14.339331 1319 leader_election.cc:302] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45fda73093f549059ff1d2f8ef9e411e, d2135c02dfb042069cfb5a9747cbff71; no voters:
I20260924 13:38:14.339418 1249 raft_consensus.cc:2454] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45fda73093f549059ff1d2f8ef9e411e in term 0.
I20260924 13:38:14.339450 1463 raft_consensus.cc:2781] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:14.339524 1463 raft_consensus.cc:482] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:14.339548 1463 raft_consensus.cc:3037] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:14.340292 1463 raft_consensus.cc:504] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.340438 1463 leader_election.cc:288] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [CANDIDATE]: Term 1 election: Requested vote from peers 0dd242ac904746a9a95f1fda67689068 (127.25.240.66:39671), d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813)
I20260924 13:38:14.340576 1249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55706974a05a4c39924418472c4b075a" candidate_uuid: "45fda73093f549059ff1d2f8ef9e411e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dd242ac904746a9a95f1fda67689068"
I20260924 13:38:14.340579 1112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55706974a05a4c39924418472c4b075a" candidate_uuid: "45fda73093f549059ff1d2f8ef9e411e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2135c02dfb042069cfb5a9747cbff71"
I20260924 13:38:14.340649 1249 raft_consensus.cc:3037] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:14.340680 1112 raft_consensus.cc:3037] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:14.341391 1112 raft_consensus.cc:2454] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45fda73093f549059ff1d2f8ef9e411e in term 1.
I20260924 13:38:14.341385 1249 raft_consensus.cc:2454] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45fda73093f549059ff1d2f8ef9e411e in term 1.
I20260924 13:38:14.341609 1319 leader_election.cc:302] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45fda73093f549059ff1d2f8ef9e411e, d2135c02dfb042069cfb5a9747cbff71; no voters:
I20260924 13:38:14.341689 1463 raft_consensus.cc:2781] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:14.341840 1463 raft_consensus.cc:686] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 1 LEADER]: Becoming Leader. State: Replica: 45fda73093f549059ff1d2f8ef9e411e, State: Running, Role: LEADER
I20260924 13:38:14.341950 1463 consensus_queue.cc:237] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } }
I20260924 13:38:14.342751 32661 catalog_manager.cc:5495] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e reported cstate change: term changed from 0 to 1, leader changed from <none> to 45fda73093f549059ff1d2f8ef9e411e (127.25.240.67). New cstate: current_term: 1 leader_uuid: "45fda73093f549059ff1d2f8ef9e411e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } health_report { overall_health: UNKNOWN } } }
I20260924 13:38:14.343389 943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: NON_VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: true } } }
I20260924 13:38:14.343495 943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:14.344045 1472 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: attempting to promote NON_VOTER 8491895e11604b73abdf8f675ebe8ef9 to VOTER
I20260924 13:38:14.344193 1472 consensus_queue.cc:237] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } }
I20260924 13:38:14.344652 803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Refusing update from remote peer 250ae62b280043d28245890fa6bbe456: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 13:38:14.344652 989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 LEARNER]: Refusing update from remote peer 250ae62b280043d28245890fa6bbe456: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 13:38:14.344882 1474 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 13:38:14.345041 755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 13:38:14.345705 1474 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8491895e11604b73abdf8f675ebe8ef9 (127.25.240.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:14.345845 803 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8491895e11604b73abdf8f675ebe8ef9 (127.25.240.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:14.345911 989 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 8491895e11604b73abdf8f675ebe8ef9 (127.25.240.124) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:14.346763 1474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:14.346840 1474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:14.346848 1018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:14.346943 1018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:38:14.346925 1474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:14.346975 1474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:14.346946 755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:14.347009 755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:14.347175 1475 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "250ae62b280043d28245890fa6bbe456" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:14.347247 1475 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:38:14.354874 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.352444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.25.240.67:36131)
0924 13:38:14.352484 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.352486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.352495 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:14.352540 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.352541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.352559 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.352642 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:14.352794 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.353275 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.354107 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.354298 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.354381 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.354553 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:14.354654 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:14.354742 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.354778 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.354799 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:14.356506 1249 raft_consensus.cc:1261] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 1 FOLLOWER]: Refusing update from remote peer 45fda73093f549059ff1d2f8ef9e411e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:14.356539 1112 raft_consensus.cc:1261] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 1 FOLLOWER]: Refusing update from remote peer 45fda73093f549059ff1d2f8ef9e411e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:14.356842 1463 consensus_queue.cc:1035] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:14.357002 1470 consensus_queue.cc:1035] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:14.359844 1478 mvcc.cc:205] Tried to move back new op lower bound from 7332893058481909760 to 7332893058425315328. Current Snapshot: MvccSnapshot[applied={T|T < 7332893058481909760}]
I20260924 13:38:14.362030 32660 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:14.362125 32660 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:14.363281 32660 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 250ae62b280043d28245890fa6bbe456: Sending DeleteTablet for 3 replicas of tablet 55706974a05a4c39924418472c4b075a
I20260924 13:38:14.363653 1092 tablet_service.cc:1514] Processing DeleteTablet for tablet 55706974a05a4c39924418472c4b075a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:14 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:44858
I20260924 13:38:14.364041 1229 tablet_service.cc:1514] Processing DeleteTablet for tablet 55706974a05a4c39924418472c4b075a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:14 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:47240
I20260924 13:38:14.364055 1365 tablet_service.cc:1514] Processing DeleteTablet for tablet 55706974a05a4c39924418472c4b075a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:14 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:33332
I20260924 13:38:14.364697 1487 tablet_replica.cc:330] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: stopping tablet replica
I20260924 13:38:14.364810 1487 raft_consensus.cc:2229] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:14.364897 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 32629
I20260924 13:38:14.364915 1487 raft_consensus.cc:2258] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:14.366525 1489 tablet_replica.cc:330] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: stopping tablet replica
I20260924 13:38:14.366631 1489 raft_consensus.cc:2229] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:38:14.366727 1489 raft_consensus.cc:2258] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:14.366760 1487 ts_tablet_manager.cc:1905] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:14.367129 1489 ts_tablet_manager.cc:1905] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:14.368441 1490 tablet_replica.cc:330] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: stopping tablet replica
I20260924 13:38:14.368774 1490 raft_consensus.cc:2229] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:14.369158 1490 raft_consensus.cc:2258] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:14.369916 1487 ts_tablet_manager.cc:1918] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:14.370007 1487 log.cc:1196] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/55706974a05a4c39924418472c4b075a
I20260924 13:38:14.369997 1490 ts_tablet_manager.cc:1905] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:14.370282 1487 ts_tablet_manager.cc:1939] T 55706974a05a4c39924418472c4b075a P d2135c02dfb042069cfb5a9747cbff71: Deleting consensus metadata
I20260924 13:38:14.371587 1490 ts_tablet_manager.cc:1918] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:14.371665 1490 log.cc:1196] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/55706974a05a4c39924418472c4b075a
I20260924 13:38:14.371918 1490 ts_tablet_manager.cc:1939] T 55706974a05a4c39924418472c4b075a P 0dd242ac904746a9a95f1fda67689068: Deleting consensus metadata
W20260924 13:38:14.372242 1183 connection.cc:441] server connection from 127.0.0.1:47240 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47240 (request call id 1) could send its response
I20260924 13:38:14.372948 1489 ts_tablet_manager.cc:1918] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:14.373023 1489 log.cc:1196] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/55706974a05a4c39924418472c4b075a
I20260924 13:38:14.373268 1489 ts_tablet_manager.cc:1939] T 55706974a05a4c39924418472c4b075a P 45fda73093f549059ff1d2f8ef9e411e: Deleting consensus metadata
W20260924 13:38:14.373632 1321 connection.cc:441] server connection from 127.0.0.1:33332 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33332 (request call id 1) could send its response
I20260924 13:38:14.373725 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.364311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49766 (local address 127.25.240.124:40639)
0924 13:38:14.364351 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.364353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.364362 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:14.364438 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.364439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.364450 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.364478 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.365497 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.366295 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.367725 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.368059 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.368100 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.370227 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.370229 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.370229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.370236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.372086 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.372624 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.372625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.372626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.372684 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.372917 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.372918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.372919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.373055 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.373138 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.373521 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.373631 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 13:38:14.374119 1046 connection.cc:441] server connection from 127.0.0.1:44858 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44858 (request call id 1) could send its response
I20260924 13:38:14.381093 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.373782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.25.240.124:40639)
0924 13:38:14.373836 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.373838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.374093 (+ 255us) server_negotiation.cc:407] Connection header received
0924 13:38:14.374188 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.374189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.374199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.374228 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.375076 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.375547 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.376276 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.376475 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.376557 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.378151 (+ 1594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.378152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.378153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.378159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.379851 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.380259 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.380260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.380261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.380320 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.380551 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.380553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.380553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.380691 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.380793 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.380926 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.380998 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":36}
I20260924 13:38:14.382112 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:14.373791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56880 (local address 127.25.240.125:46219)
0924 13:38:14.373840 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:14.373842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:14.374028 (+ 186us) server_negotiation.cc:407] Connection header received
0924 13:38:14.374129 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:14.374130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:14.374144 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:14.374174 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:14.374967 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.375739 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.376856 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:14.377141 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:14.377196 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:14.378762 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:14.378763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:14.378764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:14.378771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:14.380706 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.381262 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.381263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.381263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.381320 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:14.381540 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:14.381541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:14.381541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:14.381706 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:14.381804 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:14.381983 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 13:38:14.382053 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:38:15.361732 1497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.361387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.65:54525)
0924 13:38:15.361529 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.361648 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
W20260924 13:38:15.362022 1158 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:35077 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
I20260924 13:38:15.362285 1498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.362071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.66:42951)
0924 13:38:15.362178 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.362247 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":50,"threads_started":1}
W20260924 13:38:15.362459 1295 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:35077 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
I20260924 13:38:15.362587 1499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.362172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.67:35131)
0924 13:38:15.362443 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.362547 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":46,"threads_started":1}
W20260924 13:38:15.362752 1431 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:35077 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
I20260924 13:38:15.864024 1500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 250ae62b280043d28245890fa6bbe456)
I20260924 13:38:15.864149 1500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } }
I20260924 13:38:15.864405 1500 leader_election.cc:288] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 250ae62b280043d28245890fa6bbe456 (127.25.240.126:35077), b92a645f83a2488b9555d7c02550f35c (127.25.240.125:46219)
I20260924 13:38:15.864769 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.864521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:39168)
0924 13:38:15.864651 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.864722 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
W20260924 13:38:15.864915 964 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
I20260924 13:38:15.865248 1505 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 250ae62b280043d28245890fa6bbe456)
I20260924 13:38:15.865355 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.865262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:39180)
0924 13:38:15.865302 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.865324 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 13:38:15.865361 1505 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } }
W20260924 13:38:15.865463 964 leader_election.cc:334] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 250ae62b280043d28245890fa6bbe456 (127.25.240.126:35077): Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
I20260924 13:38:15.865814 1505 leader_election.cc:288] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 250ae62b280043d28245890fa6bbe456 (127.25.240.126:35077), 8491895e11604b73abdf8f675ebe8ef9 (127.25.240.124:40639)
I20260924 13:38:15.865932 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.865715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:39186)
0924 13:38:15.865835 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.865884 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":48,"threads_started":1}
W20260924 13:38:15.866046 778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
I20260924 13:38:15.866458 1506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.866327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:39190)
0924 13:38:15.866375 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.866431 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 13:38:15.866539 778 leader_election.cc:334] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 250ae62b280043d28245890fa6bbe456 (127.25.240.126:35077): Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
I20260924 13:38:15.868247 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.864566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:46219 (local address 127.0.0.1:56886)
0924 13:38:15.864730 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.864741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:15.864802 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:15.865250 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:15.865251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:15.865254 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:15.865317 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.865320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.866480 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.866481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.867295 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.867298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.867978 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.867986 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.868055 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.868059 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:15.868125 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:38:15.868170 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":43,"threads_started":1}
I20260924 13:38:15.868268 1503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.864664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.25.240.125:46219)
0924 13:38:15.864775 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:15.864777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:15.864789 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:38:15.865141 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:15.865142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:15.865154 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:15.865234 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:15.865497 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.866386 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.867446 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.867904 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.867967 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.867974 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:15.868144 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 13:38:15.868195 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":48,"threads_started":1}
I20260924 13:38:15.868454 803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8491895e11604b73abdf8f675ebe8ef9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b92a645f83a2488b9555d7c02550f35c" is_pre_election: true
I20260924 13:38:15.868566 803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8491895e11604b73abdf8f675ebe8ef9 in term 1.
I20260924 13:38:15.868696 963 leader_election.cc:302] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8491895e11604b73abdf8f675ebe8ef9, b92a645f83a2488b9555d7c02550f35c; no voters: 250ae62b280043d28245890fa6bbe456
I20260924 13:38:15.868805 1500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 13:38:15.868872 1500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 250ae62b280043d28245890fa6bbe456)
I20260924 13:38:15.868904 1500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:38:15.869926 1500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } }
I20260924 13:38:15.870071 1500 leader_election.cc:288] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [CANDIDATE]: Term 2 election: Requested vote from peers 250ae62b280043d28245890fa6bbe456 (127.25.240.126:35077), b92a645f83a2488b9555d7c02550f35c (127.25.240.125:46219)
I20260924 13:38:15.870100 1507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.865737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:40639 (local address 127.0.0.1:49776)
0924 13:38:15.866005 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.866015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:15.866071 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:15.866448 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:15.866449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:15.866451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:15.866493 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.866495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.868035 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.868036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.869113 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.869115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.869880 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.869881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.869927 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.869932 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:38:15.869973 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 13:38:15.870004 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":50,"threads_started":1}
I20260924 13:38:15.870121 1508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.865834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.25.240.124:40639)
0924 13:38:15.865951 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:15.865953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:15.866211 (+ 258us) server_negotiation.cc:407] Connection header received
0924 13:38:15.866235 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:15.866236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:15.866246 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:15.866420 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:15.867413 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.867966 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.869189 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.869798 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.869878 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.869888 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:15.869989 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 13:38:15.870028 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":73,"threads_started":1}
I20260924 13:38:15.870303 803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8491895e11604b73abdf8f675ebe8ef9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b92a645f83a2488b9555d7c02550f35c"
I20260924 13:38:15.870363 803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:38:15.870329 989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b92a645f83a2488b9555d7c02550f35c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8491895e11604b73abdf8f675ebe8ef9" is_pre_election: true
I20260924 13:38:15.870421 989 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b92a645f83a2488b9555d7c02550f35c in current term 2: Already voted for candidate 8491895e11604b73abdf8f675ebe8ef9 in this term.
I20260924 13:38:15.870539 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.870150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:39196)
0924 13:38:15.870480 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.870509 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
I20260924 13:38:15.870639 780 leader_election.cc:302] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b92a645f83a2488b9555d7c02550f35c; no voters: 250ae62b280043d28245890fa6bbe456, 8491895e11604b73abdf8f675ebe8ef9
I20260924 13:38:15.870874 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.870722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:39198)
0924 13:38:15.870831 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.870854 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260924 13:38:15.870951 964 leader_election.cc:334] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 250ae62b280043d28245890fa6bbe456 (127.25.240.126:35077): Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
I20260924 13:38:15.871207 803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8491895e11604b73abdf8f675ebe8ef9 in term 2.
I20260924 13:38:15.871279 1505 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:38:15.871340 963 leader_election.cc:302] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8491895e11604b73abdf8f675ebe8ef9, b92a645f83a2488b9555d7c02550f35c; no voters: 250ae62b280043d28245890fa6bbe456
I20260924 13:38:15.871440 1500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:38:15.871591 1500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [term 2 LEADER]: Becoming Leader. State: Replica: 8491895e11604b73abdf8f675ebe8ef9, State: Running, Role: LEADER
I20260924 13:38:15.871675 1500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } }
I20260924 13:38:15.871990 1510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8491895e11604b73abdf8f675ebe8ef9. Latest consensus state: current_term: 2 leader_uuid: "8491895e11604b73abdf8f675ebe8ef9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:15.872079 1510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:15.872260 1512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:15.872649 1512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:15.872707 1512 catalog_manager.cc:1259] Loaded cluster ID: 907e9bca243a4847b893d33ae34dd4b7
I20260924 13:38:15.872771 1512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:15.872931 1512 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:15.873031 1512 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Loaded TSK: 0
I20260924 13:38:15.873176 1512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:38:15.911993 979 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:38:15.913759 979 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:15.914829 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.914693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:39224)
0924 13:38:15.914729 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.914802 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 13:38:15.914844 803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [term 2 FOLLOWER]: Refusing update from remote peer 8491895e11604b73abdf8f675ebe8ef9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 13:38:15.915127 1510 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 13:38:15.915308 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.915201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.0.0.1:39240)
0924 13:38:15.915251 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.915282 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 13:38:15.915421 964 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 -> Peer 250ae62b280043d28245890fa6bbe456 (127.25.240.126:35077): Couldn't send request to peer 250ae62b280043d28245890fa6bbe456. Status: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 13:38:15.915814 1505 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8491895e11604b73abdf8f675ebe8ef9. Latest consensus state: current_term: 2 leader_uuid: "8491895e11604b73abdf8f675ebe8ef9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:15.915912 1505 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:38:15.916404 1510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8491895e11604b73abdf8f675ebe8ef9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:15.916503 1510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:15.917086 1505 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8491895e11604b73abdf8f675ebe8ef9" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "250ae62b280043d28245890fa6bbe456" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35077 } } peers { permanent_uuid: "b92a645f83a2488b9555d7c02550f35c" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 46219 } attrs { promote: false } } peers { permanent_uuid: "8491895e11604b73abdf8f675ebe8ef9" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 40639 } attrs { promote: false } } }
I20260924 13:38:15.917168 1505 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92a645f83a2488b9555d7c02550f35c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:38:15.917311 1514 mvcc.cc:205] Tried to move back new op lower bound from 7332893064864833536 to 7332893064691032064. Current Snapshot: MvccSnapshot[applied={T|T < 7332893064864833536 or (T in {7332893064864833536})}]
I20260924 13:38:15.921535 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.918220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:39671 (local address 127.0.0.1:47242)
0924 13:38:15.918291 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.918302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:15.918351 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:15.918849 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:15.918850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:15.918852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:15.918896 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.918898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.919545 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.919546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.920654 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.920656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.921301 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.921301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.921360 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.921364 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:15.921414 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:38:15.921462 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":3}
I20260924 13:38:15.921626 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.918199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:43813 (local address 127.0.0.1:44864)
0924 13:38:15.918257 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.918267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:15.918314 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:15.918736 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:15.918737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:15.918740 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:15.918803 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.918804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.919782 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.919783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.920577 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.920580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.921353 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.921354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.921412 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.921415 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:15.921462 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:38:15.921504 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":35}
I20260924 13:38:15.921548 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.918608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47242 (local address 127.25.240.66:39671)
0924 13:38:15.918711 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:15.918713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:15.918721 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:15.918749 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:15.918750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:15.918760 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:15.918964 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:15.919016 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.919461 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.920748 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.921212 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.921293 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.921304 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:15.921421 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 13:38:15.921466 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":60,"threads_started":1}
I20260924 13:38:15.921881 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.918294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:36131 (local address 127.0.0.1:33338)
0924 13:38:15.918428 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.918438 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:15.918482 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:15.918930 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:15.918931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:15.918933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:15.918962 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.918963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.919814 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.919815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.920882 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:15.920883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.921620 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.921621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:15.921701 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.921707 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:38:15.921741 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 13:38:15.921786 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260924 13:38:15.922067 1229 tablet_service.cc:1467] Processing CreateTablet for tablet c0c272acbe5a431caec96731bfee1984 (DEFAULT_TABLE table=test-table [id=4cd0b3e9d97c400ca8fa2c9302d1ee82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:15.922139 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.918481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.25.240.67:36131)
0924 13:38:15.918798 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:15.918800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:15.918807 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:15.918829 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:15.918830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:15.918840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:15.919017 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:15.919049 (+ 32us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.919736 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.920959 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.921508 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.921994 (+ 486us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.922009 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:15.922031 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 13:38:15.922074 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":66,"threads_started":1}
I20260924 13:38:15.922191 1229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c272acbe5a431caec96731bfee1984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:15.922367 1365 tablet_service.cc:1467] Processing CreateTablet for tablet c0c272acbe5a431caec96731bfee1984 (DEFAULT_TABLE table=test-table [id=4cd0b3e9d97c400ca8fa2c9302d1ee82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:15.922470 1365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c272acbe5a431caec96731bfee1984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:15.923722 1522 tablet_bootstrap.cc:492] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: Bootstrap starting.
I20260924 13:38:15.923780 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.918452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.25.240.65:43813)
0924 13:38:15.918599 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:15.918601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:15.918609 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:15.918639 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:15.918640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:15.918648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:15.918719 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:15.918884 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.919676 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.920672 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:15.921256 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:15.921351 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:15.921362 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:15.921840 (+ 478us) server_negotiation.cc:299] Negotiation successful
0924 13:38:15.921906 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260924 13:38:15.923972 1092 tablet_service.cc:1467] Processing CreateTablet for tablet c0c272acbe5a431caec96731bfee1984 (DEFAULT_TABLE table=test-table [id=4cd0b3e9d97c400ca8fa2c9302d1ee82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:15.923978 1523 tablet_bootstrap.cc:492] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: Bootstrap starting.
I20260924 13:38:15.924089 1092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c272acbe5a431caec96731bfee1984. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:15.924355 1522 tablet_bootstrap.cc:654] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:15.924355 1523 tablet_bootstrap.cc:654] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:15.924988 1523 tablet_bootstrap.cc:492] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: No bootstrap required, opened a new log
I20260924 13:38:15.925055 1523 ts_tablet_manager.cc:1397] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:15.925176 1523 raft_consensus.cc:348] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:15.925264 1523 raft_consensus.cc:374] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:15.925257 1526 tablet_bootstrap.cc:492] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71: Bootstrap starting.
I20260924 13:38:15.925292 1523 raft_consensus.cc:729] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fda73093f549059ff1d2f8ef9e411e, State: Initialized, Role: FOLLOWER
I20260924 13:38:15.925333 1523 consensus_queue.cc:260] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:15.925493 1523 ts_tablet_manager.cc:1428] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:15.925652 1526 tablet_bootstrap.cc:654] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:15.926250 1526 tablet_bootstrap.cc:492] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71: No bootstrap required, opened a new log
I20260924 13:38:15.926235 1432 heartbeater.cc:503] Master 127.25.240.124:40639 was elected leader, sending a full tablet report...
I20260924 13:38:15.926331 1526 ts_tablet_manager.cc:1397] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:38:15.926393 1522 tablet_bootstrap.cc:492] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: No bootstrap required, opened a new log
I20260924 13:38:15.926453 1522 ts_tablet_manager.cc:1397] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:38:15.926529 1526 raft_consensus.cc:348] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:15.926610 1526 raft_consensus.cc:374] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:15.926642 1526 raft_consensus.cc:729] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2135c02dfb042069cfb5a9747cbff71, State: Initialized, Role: FOLLOWER
I20260924 13:38:15.926606 1522 raft_consensus.cc:348] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:15.926676 1522 raft_consensus.cc:374] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:15.926707 1522 raft_consensus.cc:729] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dd242ac904746a9a95f1fda67689068, State: Initialized, Role: FOLLOWER
I20260924 13:38:15.926759 1526 consensus_queue.cc:260] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:15.926817 1522 consensus_queue.cc:260] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:15.926926 1526 ts_tablet_manager.cc:1428] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:38:15.927035 1522 ts_tablet_manager.cc:1428] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:15.927384 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.925683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.67:59341)
0924 13:38:15.926991 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.927031 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":74,"threads_started":1}
I20260924 13:38:15.927824 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.927739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.67:37429)
0924 13:38:15.927774 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.927798 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 13:38:15.927871 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.927119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.65:50107)
0924 13:38:15.927781 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.927827 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":72,"threads_started":1}
I20260924 13:38:15.928030 1159 heartbeater.cc:503] Master 127.25.240.124:40639 was elected leader, sending a full tablet report...
I20260924 13:38:15.928196 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.927220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.66:56481)
0924 13:38:15.928130 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.928166 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":105,"threads_started":1}
I20260924 13:38:15.928270 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.928113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.65:43097)
0924 13:38:15.928217 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.928242 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 13:38:15.928570 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:15.928412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.66:54811)
0924 13:38:15.928521 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:15.928545 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
W20260924 13:38:15.933658 1434 tablet.cc:2369] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:38:16.014173 1531 raft_consensus.cc:482] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:16.014267 1531 raft_consensus.cc:504] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:16.014629 1531 leader_election.cc:288] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813), 45fda73093f549059ff1d2f8ef9e411e (127.25.240.67:36131)
I20260924 13:38:16.018615 1520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.014996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:54647 (local address 127.25.240.67:36131)
0924 13:38:16.015058 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:16.015060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:16.015068 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:16.015091 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:16.015092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:16.015100 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:16.015157 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:16.015363 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.016059 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.017681 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.018230 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.018336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:16.018347 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:16.018507 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 13:38:16.018544 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:38:16.018615 1519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.014861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:50829 (local address 127.25.240.65:43813)
0924 13:38:16.014943 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:16.014946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:16.014958 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:38:16.014989 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:16.014990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:16.015003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:16.015064 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:16.015236 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.016037 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.017681 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.018230 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.018336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:16.018347 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:16.018508 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 13:38:16.018544 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 13:38:16.018656 1533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.014774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:43813 (local address 127.25.240.66:50829)
0924 13:38:16.014839 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.014853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:16.014917 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:16.015070 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:16.015071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:16.015074 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:16.015173 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:16.015175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.016127 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.016128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:16.017509 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:16.017511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.018354 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.018354 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:16.018422 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:16.018424 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:38:16.018482 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 13:38:16.018539 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":64}
I20260924 13:38:16.018658 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.014923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:36131 (local address 127.25.240.66:54647)
0924 13:38:16.014977 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.014986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:16.015030 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:16.015201 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:16.015202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:16.015204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:16.015280 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:16.015281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.016129 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.016130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:16.017510 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:16.017511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.018337 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.018338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:16.018413 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:16.018418 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:38:16.018483 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 13:38:16.018539 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":103}
I20260924 13:38:16.019012 1385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c272acbe5a431caec96731bfee1984" candidate_uuid: "0dd242ac904746a9a95f1fda67689068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45fda73093f549059ff1d2f8ef9e411e" is_pre_election: true
I20260924 13:38:16.019093 1112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c272acbe5a431caec96731bfee1984" candidate_uuid: "0dd242ac904746a9a95f1fda67689068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2135c02dfb042069cfb5a9747cbff71" is_pre_election: true
I20260924 13:38:16.019141 1385 raft_consensus.cc:2454] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dd242ac904746a9a95f1fda67689068 in term 0.
I20260924 13:38:16.019163 1112 raft_consensus.cc:2454] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0dd242ac904746a9a95f1fda67689068 in term 0.
I20260924 13:38:16.019304 1183 leader_election.cc:302] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dd242ac904746a9a95f1fda67689068, d2135c02dfb042069cfb5a9747cbff71; no voters:
I20260924 13:38:16.019450 1531 raft_consensus.cc:2781] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:16.019500 1531 raft_consensus.cc:482] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:16.019527 1531 raft_consensus.cc:3037] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:16.020172 1531 raft_consensus.cc:504] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:16.020309 1531 leader_election.cc:288] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [CANDIDATE]: Term 1 election: Requested vote from peers d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813), 45fda73093f549059ff1d2f8ef9e411e (127.25.240.67:36131)
I20260924 13:38:16.020469 1112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c272acbe5a431caec96731bfee1984" candidate_uuid: "0dd242ac904746a9a95f1fda67689068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2135c02dfb042069cfb5a9747cbff71"
I20260924 13:38:16.020489 1385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c272acbe5a431caec96731bfee1984" candidate_uuid: "0dd242ac904746a9a95f1fda67689068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45fda73093f549059ff1d2f8ef9e411e"
I20260924 13:38:16.020550 1112 raft_consensus.cc:3037] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:16.020573 1385 raft_consensus.cc:3037] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:16.021132 1112 raft_consensus.cc:2454] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dd242ac904746a9a95f1fda67689068 in term 1.
I20260924 13:38:16.021288 1385 raft_consensus.cc:2454] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0dd242ac904746a9a95f1fda67689068 in term 1.
I20260924 13:38:16.021289 1183 leader_election.cc:302] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dd242ac904746a9a95f1fda67689068, d2135c02dfb042069cfb5a9747cbff71; no voters:
I20260924 13:38:16.021458 1531 raft_consensus.cc:2781] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:16.021665 1531 raft_consensus.cc:686] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 1 LEADER]: Becoming Leader. State: Replica: 0dd242ac904746a9a95f1fda67689068, State: Running, Role: LEADER
I20260924 13:38:16.021765 1531 consensus_queue.cc:237] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } }
I20260924 13:38:16.021773 1296 heartbeater.cc:503] Master 127.25.240.124:40639 was elected leader, sending a full tablet report...
I20260924 13:38:16.022002 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.021869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.66:51569)
0924 13:38:16.021936 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.021964 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 13:38:16.022491 978 catalog_manager.cc:5495] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0dd242ac904746a9a95f1fda67689068 (127.25.240.66). New cstate: current_term: 1 leader_uuid: "0dd242ac904746a9a95f1fda67689068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0dd242ac904746a9a95f1fda67689068" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39671 } health_report { overall_health: HEALTHY } } }
I20260924 13:38:16.037739 1521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.035348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.25.240.66:39671)
0924 13:38:16.035392 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:16.035394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:16.035411 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:38:16.035456 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:16.035457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:16.035474 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:16.035558 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:16.035675 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.036135 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.036894 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:16.037088 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:16.037164 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:16.037344 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:16.037443 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:16.037538 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:16.037629 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 13:38:16.037650 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:38:16.038792 1112 raft_consensus.cc:1261] T c0c272acbe5a431caec96731bfee1984 P d2135c02dfb042069cfb5a9747cbff71 [term 1 FOLLOWER]: Refusing update from remote peer 0dd242ac904746a9a95f1fda67689068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:16.038925 1385 raft_consensus.cc:1261] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 1 FOLLOWER]: Refusing update from remote peer 0dd242ac904746a9a95f1fda67689068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:16.039047 1535 consensus_queue.cc:1035] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2135c02dfb042069cfb5a9747cbff71" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:16.039209 1531 consensus_queue.cc:1035] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45fda73093f549059ff1d2f8ef9e411e" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 36131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:16.039744 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.039592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.65:56335)
0924 13:38:16.039657 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.039714 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 13:38:16.039985 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.039883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.67:59459)
0924 13:38:16.039926 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.039954 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 13:38:16.040712 1538 mvcc.cc:205] Tried to move back new op lower bound from 7332893065372524544 to 7332893065305600000. Current Snapshot: MvccSnapshot[applied={T|T < 7332893065372524544}]
I20260924 13:38:16.041167 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.040946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.66:46505)
0924 13:38:16.041108 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.041138 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260924 13:38:16.041590 1532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.041482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.65:51793)
0924 13:38:16.041511 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.041543 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 13:38:16.042234 1528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.042151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.67:59715)
0924 13:38:16.042182 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.042208 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 13:38:16.042706 1534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:16.042596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35077 (local address 127.25.240.66:54903)
0924 13:38:16.042652 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.042677 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:35077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 13:38:16.044065 978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:16.044159 978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:16.045531 978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8491895e11604b73abdf8f675ebe8ef9: Sending DeleteTablet for 3 replicas of tablet c0c272acbe5a431caec96731bfee1984
I20260924 13:38:16.045931 1229 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c272acbe5a431caec96731bfee1984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:16 UTC) from {username='oryx', principal='oryx/127.25.240.124@KRBTEST.COM'} at 127.0.0.1:47242
I20260924 13:38:16.046118 1549 tablet_replica.cc:330] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: stopping tablet replica
I20260924 13:38:16.046113 1092 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c272acbe5a431caec96731bfee1984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:16 UTC) from {username='oryx', principal='oryx/127.25.240.124@KRBTEST.COM'} at 127.0.0.1:44864
I20260924 13:38:16.046180 1549 raft_consensus.cc:2229] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:38:16.046264 1549 raft_consensus.cc:2258] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:16.046319 1365 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c272acbe5a431caec96731bfee1984 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:16 UTC) from {username='oryx', principal='oryx/127.25.240.124@KRBTEST.COM'} at 127.0.0.1:33338
I20260924 13:38:16.046350 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 940
I20260924 13:38:16.046867 1551 tablet_replica.cc:330] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: stopping tablet replica
I20260924 13:38:16.046917 1551 raft_consensus.cc:2229] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:16.046959 1551 raft_consensus.cc:2258] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:16.047224 1551 ts_tablet_manager.cc:1905] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:16.048929 1551 ts_tablet_manager.cc:1918] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:16.048980 1551 log.cc:1196] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/c0c272acbe5a431caec96731bfee1984
I20260924 13:38:16.049227 1551 ts_tablet_manager.cc:1939] T c0c272acbe5a431caec96731bfee1984 P 45fda73093f549059ff1d2f8ef9e411e: Deleting consensus metadata
I20260924 13:38:16.049533 964 catalog_manager.cc:4841] TS 45fda73093f549059ff1d2f8ef9e411e (127.25.240.67:36131): tablet c0c272acbe5a431caec96731bfee1984 (table test-table [id=4cd0b3e9d97c400ca8fa2c9302d1ee82]) successfully deleted
I20260924 13:38:16.050693 1549 ts_tablet_manager.cc:1905] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:16.052577 1549 ts_tablet_manager.cc:1918] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:16.052634 1549 log.cc:1196] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/c0c272acbe5a431caec96731bfee1984
I20260924 13:38:16.052875 1549 ts_tablet_manager.cc:1939] T c0c272acbe5a431caec96731bfee1984 P 0dd242ac904746a9a95f1fda67689068: Deleting consensus metadata
I20260924 13:38:16.053622 963 catalog_manager.cc:4841] TS 0dd242ac904746a9a95f1fda67689068 (127.25.240.66:39671): tablet c0c272acbe5a431caec96731bfee1984 (table test-table [id=4cd0b3e9d97c400ca8fa2c9302d1ee82]) successfully deleted
W20260924 13:38:16.053901 1518 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:16.053323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:43813 (local address 127.0.0.1:44872)
0924 13:38:16.053387 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:16.053401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:16.053447 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:16.053856 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:43813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 13:38:16.054036 964 catalog_manager.cc:4584] TS d2135c02dfb042069cfb5a9747cbff71 (127.25.240.65:43813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c0c272acbe5a431caec96731bfee1984: Network error: Client connection negotiation failed: client connection to 127.25.240.65:43813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:38:16.054293 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 1166
I20260924 13:38:16.059028 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 1302
I20260924 13:38:16.064685 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 658
I20260924 13:38:16.069432 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 847
2026-09-24T13:38:16Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4584 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:16 dist-test-slave-m5z1 krb5kdc[1554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:16 dist-test-slave-m5z1 krb5kdc[1554](info): set up 2 sockets
Sep 24 13:38:16 dist-test-slave-m5z1 krb5kdc[1554](info): commencing operation
krb5kdc: starting...
W20260924 13:38:17.122278 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.126.keytab.
Entry for principal oryx/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.126.keytab.
I20260924 13:38:17.236799 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:46827
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.126.keytab
--principal=oryx/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:17.316071 1567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:17.316300 1567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:17.316337 1567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:17.317481 1567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:17.317544 1567 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 13:38:17.317565 1567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:17.317605 1567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:17.317633 1567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:17.317659 1567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:17.319141 1567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:46827
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.126.keytab
--principal=oryx/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:17.319407 1567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:17.319617 1567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:17.324325 1572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:17.324321 1573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:17.324432 1575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:17.324321 1567 server_base.cc:1034] running on GCE node
I20260924 13:38:17.324848 1567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:17.325178 1567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:17.326339 1567 hybrid_clock.cc:648] HybridClock initialized: now 1790257097326326 us; error 39 us; skew 500 ppm
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:17.328728 1567 init.cc:377] Logged in from keytab as oryx/127.25.240.126@KRBTEST.COM (short username oryx)
I20260924 13:38:17.329191 1567 webserver.cc:460] Webserver started at http://127.25.240.126:33231/ using document root <none> and password file <none>
I20260924 13:38:17.329414 1567 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:17.329483 1567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:17.329651 1567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:17.330663 1567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "2d2c22ee5eb341208c05ede4fefe1c30"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "9371a3a30f89d1dbe16ee8b60412c47c"
server_key_iv: "10bb785c37654fd88991459c7077c4e8"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.331033 1567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "2d2c22ee5eb341208c05ede4fefe1c30"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "9371a3a30f89d1dbe16ee8b60412c47c"
server_key_iv: "10bb785c37654fd88991459c7077c4e8"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.332307 1567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:38:17.333019 1582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:17.333189 1567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:17.333287 1567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "2d2c22ee5eb341208c05ede4fefe1c30"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "9371a3a30f89d1dbe16ee8b60412c47c"
server_key_iv: "10bb785c37654fd88991459c7077c4e8"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.333365 1567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:17.352371 1567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:17.354700 1567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:17.354897 1567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:17.359606 1567 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:46827
I20260924 13:38:17.359653 1644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:46827 every 8 connection(s)
I20260924 13:38:17.359968 1567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:17.360582 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:17.361001 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 1567
I20260924 13:38:17.361121 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:17.361328 26561 external_mini_cluster.cc:1428] Setting key b95b898925a3fbf1cb44c29c2e38ee56
I20260924 13:38:17.363389 1645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: Bootstrap starting.
I20260924 13:38:17.363996 1645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:17.364316 1645 log.cc:824] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:17.364985 1645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: No bootstrap required, opened a new log
I20260924 13:38:17.366851 1645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2c22ee5eb341208c05ede4fefe1c30" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46827 } }
I20260924 13:38:17.367002 1645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:17.367027 1645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2c22ee5eb341208c05ede4fefe1c30, State: Initialized, Role: FOLLOWER
I20260924 13:38:17.367163 1645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2c22ee5eb341208c05ede4fefe1c30" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46827 } }
I20260924 13:38:17.367235 1645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:17.367295 1645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:17.367343 1645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:17.368060 1645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2c22ee5eb341208c05ede4fefe1c30" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46827 } }
I20260924 13:38:17.368173 1645 leader_election.cc:302] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d2c22ee5eb341208c05ede4fefe1c30; no voters:
I20260924 13:38:17.368341 1645 leader_election.cc:288] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:17.368393 1650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:17.368564 1650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2c22ee5eb341208c05ede4fefe1c30, State: Running, Role: LEADER
I20260924 13:38:17.368623 1645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:17.368701 1650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2c22ee5eb341208c05ede4fefe1c30" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46827 } }
I20260924 13:38:17.369050 1652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d2c22ee5eb341208c05ede4fefe1c30. Latest consensus state: current_term: 1 leader_uuid: "2d2c22ee5eb341208c05ede4fefe1c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2c22ee5eb341208c05ede4fefe1c30" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46827 } } }
I20260924 13:38:17.369153 1652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:17.369134 1651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d2c22ee5eb341208c05ede4fefe1c30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d2c22ee5eb341208c05ede4fefe1c30" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46827 } } }
I20260924 13:38:17.369861 1651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30 [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:17.371160 1659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:17.371711 1659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:17.372154 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.361908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.25.240.126:46827)
0924 13:38:17.362092 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.362102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.362139 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:38:17.362549 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.362555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.362598 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.362750 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.363376 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.364446 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.365247 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.365733 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.365861 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.368504 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:17.368508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:17.368509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:17.368520 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:17.370407 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.370848 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.370851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.370851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.370970 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.371203 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.371204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.371204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.371534 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:17.371677 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.371833 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.371967 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260924 13:38:17.373453 1659 catalog_manager.cc:1347] Generated new cluster ID: 4904bbfee1004fe6ac86f0b257887e07
I20260924 13:38:17.373510 1659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:17.387075 1659 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:17.387590 1659 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:17.397311 1659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: Generated new TSK 0
I20260924 13:38:17.397524 1659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab.
Entry for principal oryx/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab.
I20260924 13:38:17.424301 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46827
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab
--principal=oryx/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:17.504287 1671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:17.504550 1671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:17.504575 1671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:17.505745 1671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:17.505789 1671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:17.505810 1671 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:38:17.505864 1671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:17.507346 1671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab
--principal=oryx/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:17.507601 1671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:17.507830 1671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:17.512795 1676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:17.512786 1679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:17.512795 1677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:17.513046 1671 server_base.cc:1034] running on GCE node
I20260924 13:38:17.513260 1671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:17.513484 1671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:17.514658 1671 hybrid_clock.cc:648] HybridClock initialized: now 1790257097514625 us; error 37 us; skew 500 ppm
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:17.517115 1671 init.cc:377] Logged in from keytab as oryx/127.25.240.65@KRBTEST.COM (short username oryx)
I20260924 13:38:17.517623 1671 webserver.cc:460] Webserver started at http://127.25.240.65:45913/ using document root <none> and password file <none>
I20260924 13:38:17.517853 1671 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:17.517897 1671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:17.518059 1671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:17.519134 1671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "f0dcffeabdb1472297753670bf8bd10d"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "6c65d08cfbb8583a9f2ae549eb3d3ff3"
server_key_iv: "99399386665f0f55244fc53f8300354c"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.519476 1671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "f0dcffeabdb1472297753670bf8bd10d"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "6c65d08cfbb8583a9f2ae549eb3d3ff3"
server_key_iv: "99399386665f0f55244fc53f8300354c"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.520723 1671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:17.521406 1686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:17.521637 1671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:17.521724 1671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "f0dcffeabdb1472297753670bf8bd10d"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "6c65d08cfbb8583a9f2ae549eb3d3ff3"
server_key_iv: "99399386665f0f55244fc53f8300354c"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.521796 1671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:17.538091 1671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:17.540463 1671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:17.540640 1671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:17.541206 1671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:17.541266 1671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:17.541309 1671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:17.541338 1671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:17.548354 1671 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:40055
I20260924 13:38:17.548420 1801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:40055 every 8 connection(s)
I20260924 13:38:17.548776 1671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:38:17.549350 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 1671
I20260924 13:38:17.549469 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:17.549736 26561 external_mini_cluster.cc:1428] Setting key 464ffaa6d1927210b500cf63c11715d9
I20260924 13:38:17.551434 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.541601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:38663 (local address 127.25.240.126:46827)
0924 13:38:17.541653 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.541656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.541902 (+ 246us) server_negotiation.cc:407] Connection header received
0924 13:38:17.542389 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.542391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.542405 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.542486 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.543151 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.543694 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.545265 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.545488 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.545565 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.548274 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:17.548276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:17.548276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:17.548284 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:17.550064 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.550448 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.550449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.550450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.550509 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.550780 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.550781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.550782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.550902 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:17.550974 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.551260 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.551343 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:38:17.551606 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.541606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46827 (local address 127.25.240.65:38663)
0924 13:38:17.541756 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.541796 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.542289 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.542562 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.542566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.542798 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:17.543047 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.543052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.543794 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.543796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.545169 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.545172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.545632 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.545633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.545765 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.546296 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:17.546305 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:17.548167 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:17.550141 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.550148 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.550150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.550369 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.550593 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.550594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.550594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.550710 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.550976 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:17.550983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:17.551125 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.551227 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.551379 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260924 13:38:17.552074 1802 heartbeater.cc:347] Connected to a master server at 127.25.240.126:46827
I20260924 13:38:17.552160 1802 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:17.552356 1802 heartbeater.cc:511] Master 127.25.240.126:46827 requested a full tablet report, sending...
I20260924 13:38:17.552868 1599 ts_manager.cc:194] Registered new tserver with Master: f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055)
I20260924 13:38:17.553820 1599 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:38663
WARNING: no policy specified for oryx/127.25.240.66@KRBTEST.COM; defaulting to no policy
I20260924 13:38:17.559062 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.553687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:33013 (local address 127.25.240.126:46827)
0924 13:38:17.553728 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.553730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.553737 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:17.553809 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.553809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.553818 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.553861 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.554255 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.554763 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.555477 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.555679 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.555762 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.556634 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:17.556635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:17.556635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:17.556641 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:17.558007 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.558366 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.558367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.558368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.558444 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.558681 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.558681 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.558681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.558774 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:17.558824 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.558944 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.558996 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:38:17.559068 1809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.553560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46827 (local address 127.25.240.65:33013)
0924 13:38:17.553687 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.553698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.553751 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.553863 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.553863 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.554085 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:17.554160 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.554162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.554862 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.554863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.555378 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.555379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.555764 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.555765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.555839 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.556271 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:17.556274 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:17.556539 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:17.558113 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.558114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.558114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.558281 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.558527 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.558528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.558529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.558605 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.558835 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:17.558836 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:17.558874 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.558928 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.559007 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab.
Entry for principal oryx/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab.
I20260924 13:38:17.576004 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46827
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab
--principal=oryx/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:17.654418 1811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:17.654613 1811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:17.654637 1811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:17.655772 1811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:17.655815 1811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:17.655836 1811 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:38:17.655897 1811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:17.657346 1811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab
--principal=oryx/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:17.657601 1811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:17.657822 1811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:17.662635 1817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:17.662694 1811 server_base.cc:1034] running on GCE node
W20260924 13:38:17.662624 1819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:17.662604 1816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:17.663014 1811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:17.663285 1811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:17.664438 1811 hybrid_clock.cc:648] HybridClock initialized: now 1790257097664413 us; error 35 us; skew 500 ppm
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:17.666929 1811 init.cc:377] Logged in from keytab as oryx/127.25.240.66@KRBTEST.COM (short username oryx)
I20260924 13:38:17.667377 1811 webserver.cc:460] Webserver started at http://127.25.240.66:43261/ using document root <none> and password file <none>
I20260924 13:38:17.667591 1811 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:17.667658 1811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:17.667804 1811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:17.668875 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "8f1ae328ed834eda806797a52459a79f"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "1476ee38c0629a30ced4c5edf80e2a45"
server_key_iv: "60386402f6b02f1fd95333906c9aa882"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.669242 1811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "8f1ae328ed834eda806797a52459a79f"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "1476ee38c0629a30ced4c5edf80e2a45"
server_key_iv: "60386402f6b02f1fd95333906c9aa882"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.670651 1811 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:38:17.671344 1826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:17.671512 1811 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:17.671614 1811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "8f1ae328ed834eda806797a52459a79f"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "1476ee38c0629a30ced4c5edf80e2a45"
server_key_iv: "60386402f6b02f1fd95333906c9aa882"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.671685 1811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:17.683008 1811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:17.685194 1811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:17.685346 1811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:17.685978 1811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:17.686048 1811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:17.686079 1811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:17.686097 1811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:17.692961 1811 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:42093
I20260924 13:38:17.693037 1941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:42093 every 8 connection(s)
I20260924 13:38:17.693331 1811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:38:17.695822 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.686335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:45365 (local address 127.25.240.126:46827)
0924 13:38:17.686404 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.686407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.686571 (+ 164us) server_negotiation.cc:407] Connection header received
0924 13:38:17.687105 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.687107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.687120 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.687177 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.687836 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.688668 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.690370 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.690593 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.690688 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.693355 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:17.693357 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:17.693357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:17.693366 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:17.694726 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.695087 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.695088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.695088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.695139 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.695346 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.695347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.695347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.695444 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:17.695493 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.695701 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.695757 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:38:17.695979 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.686301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46827 (local address 127.25.240.66:45365)
0924 13:38:17.686439 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.686487 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.686977 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.687245 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.687249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.687555 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:17.687736 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.687741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.688777 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.688778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.690270 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.690275 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.690709 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.690727 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.690860 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.691744 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:17.691751 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:17.693254 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:17.694797 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.694802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.694804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.695013 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.695202 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.695203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.695203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.695293 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.695497 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:17.695500 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:17.695605 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.695689 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.695802 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260924 13:38:17.696453 1942 heartbeater.cc:347] Connected to a master server at 127.25.240.126:46827
I20260924 13:38:17.696535 1942 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:17.696723 1942 heartbeater.cc:511] Master 127.25.240.126:46827 requested a full tablet report, sending...
I20260924 13:38:17.697137 1598 ts_manager.cc:194] Registered new tserver with Master: 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093)
I20260924 13:38:17.697667 1598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:45365
I20260924 13:38:17.700428 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 1811
I20260924 13:38:17.700567 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:17.700819 26561 external_mini_cluster.cc:1428] Setting key 3e5cc412ea48b01ae4feefc7d224006f
I20260924 13:38:17.703419 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.697519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:50385 (local address 127.25.240.126:46827)
0924 13:38:17.697574 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.697576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.697636 (+ 60us) server_negotiation.cc:407] Connection header received
0924 13:38:17.697704 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.697705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.697712 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.697784 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.698084 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.698546 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.699242 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.699447 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.699520 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.700282 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:17.700283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:17.700283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:17.700290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:17.701978 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.702424 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.702425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.702426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.702504 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.702744 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.702744 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.702745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.703002 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:17.703073 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.703216 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.703308 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:17.703425 1948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.697419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46827 (local address 127.25.240.66:50385)
0924 13:38:17.697580 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.697592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.697650 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.697788 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.697789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.697959 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:17.698004 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.698005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.698637 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.698638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.699156 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.699158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.699522 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.699523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.699593 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.699929 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:17.699933 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:17.700197 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:17.702073 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.702074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.702074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.702337 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.702588 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.702589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.702589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.702681 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.703075 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:17.703076 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:17.703113 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.703169 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.703299 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab.
Entry for principal oryx/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab.
I20260924 13:38:17.725929 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46827
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab
--principal=oryx/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:17.806442 1951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:17.806664 1951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:17.806701 1951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:17.807873 1951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:17.807936 1951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:17.807968 1951 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 13:38:17.808010 1951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:17.809505 1951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab
--principal=oryx/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.240.126:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:17.809789 1951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:17.810029 1951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:17.815155 1957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:17.815197 1959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:17.815145 1956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:17.815372 1951 server_base.cc:1034] running on GCE node
I20260924 13:38:17.815591 1951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:17.815840 1951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:17.817024 1951 hybrid_clock.cc:648] HybridClock initialized: now 1790257097816996 us; error 53 us; skew 500 ppm
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:17.823220 1951 init.cc:377] Logged in from keytab as oryx/127.25.240.67@KRBTEST.COM (short username oryx)
I20260924 13:38:17.823747 1951 webserver.cc:460] Webserver started at http://127.25.240.67:44375/ using document root <none> and password file <none>
I20260924 13:38:17.823984 1951 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:17.824028 1951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:17.824162 1951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:17.825253 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "bf2df2750c104dbd9f38deee5be88a99"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "732ad140699816d325bb2eda478a0923"
server_key_iv: "79a93aa1ff57596a659d48b97bf77baf"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.825676 1951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "bf2df2750c104dbd9f38deee5be88a99"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "732ad140699816d325bb2eda478a0923"
server_key_iv: "79a93aa1ff57596a659d48b97bf77baf"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.826936 1951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:17.827620 1966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:17.827919 1951 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:17.828011 1951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "bf2df2750c104dbd9f38deee5be88a99"
format_stamp: "Formatted at 2026-09-24 13:38:17 on dist-test-slave-m5z1"
server_key: "732ad140699816d325bb2eda478a0923"
server_key_iv: "79a93aa1ff57596a659d48b97bf77baf"
server_key_version: "encryptionkey@0"
I20260924 13:38:17.828094 1951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:17.846762 1951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:17.848474 1951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:17.848637 1951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:17.849211 1951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:17.849259 1951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:17.849289 1951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:17.849308 1951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:17.856691 1951 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:40055
I20260924 13:38:17.856770 2081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:40055 every 8 connection(s)
I20260924 13:38:17.857108 1951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:38:17.859162 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.849608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:36015 (local address 127.25.240.126:46827)
0924 13:38:17.849660 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.849663 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.849927 (+ 264us) server_negotiation.cc:407] Connection header received
0924 13:38:17.850629 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.850630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.850640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.850683 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.851577 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.852132 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.853844 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.854065 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.854145 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.856567 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:17.856568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:17.856569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:17.856577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:17.858068 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.858427 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.858427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.858428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.858481 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.858683 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.858684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.858684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.858781 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:17.858841 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.859041 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.859102 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:38:17.859411 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.849620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46827 (local address 127.25.240.67:36015)
0924 13:38:17.849777 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.849827 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.850525 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.850763 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.850768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.851163 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:17.851466 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.851473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.852233 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.852235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.853766 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.853770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.854176 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.854178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.854310 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.854798 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:17.854807 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:17.856456 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:17.858139 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.858144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.858145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.858353 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.858539 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.858540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.858540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.858630 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.858831 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:17.858834 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:17.858935 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.859010 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.859200 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
I20260924 13:38:17.859869 2082 heartbeater.cc:347] Connected to a master server at 127.25.240.126:46827
I20260924 13:38:17.859948 2082 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:17.860136 2082 heartbeater.cc:511] Master 127.25.240.126:46827 requested a full tablet report, sending...
I20260924 13:38:17.860498 1599 ts_manager.cc:194] Registered new tserver with Master: bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055)
I20260924 13:38:17.860510 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 1951
I20260924 13:38:17.860625 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:17.860875 26561 external_mini_cluster.cc:1428] Setting key 5900fb6a43b23cf90f9104f06da02309
I20260924 13:38:17.860977 1599 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:36015
I20260924 13:38:17.861726 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:38:17.866973 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.860875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46827 (local address 127.25.240.67:38919)
0924 13:38:17.860992 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.861003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.861091 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.861395 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.861396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.861617 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:17.861683 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.861685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.862557 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.862558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.863044 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.863046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.863521 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.863522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.863594 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.863959 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:17.863964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:17.864352 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:17.865978 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.865979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.865980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.866204 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.866388 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:17.866389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:17.866389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:17.866459 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:17.866743 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:17.866744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:17.866779 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.866835 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.866907 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
I20260924 13:38:17.867121 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.860985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:38919 (local address 127.25.240.126:46827)
0924 13:38:17.861023 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.861025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.861065 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:38:17.861292 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.861293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.861303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.861356 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.861785 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.862469 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.863126 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.863437 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.863509 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.864443 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:17.864444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:17.864444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:17.864450 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:17.865903 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.866280 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.866281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.866281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.866327 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.866526 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.866527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.866528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.866676 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:17.866747 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.866890 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.866968 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:38:17.882964 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:17.878087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.25.240.126:46827)
0924 13:38:17.878128 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.878130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.878137 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:17.878185 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.878186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.878193 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.878237 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.878731 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.879185 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.879873 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.880043 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.880088 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.882894 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45172: BlockingRecv error: recv got EOF from 127.0.0.1:45172 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 13:38:17 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:17.890184 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.883845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.25.240.126:46827)
0924 13:38:17.883902 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.883904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.883987 (+ 83us) server_negotiation.cc:407] Connection header received
0924 13:38:17.884060 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.884062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.884070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.884138 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:17.884599 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.885043 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.885800 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.885982 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.886073 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.887971 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:17.887972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:17.887972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:17.887978 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:17.889310 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.889644 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.889644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.889645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.889694 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:17.889841 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:17.889841 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:17.889841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:17.889923 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:17.889989 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.890076 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.890128 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:38:17.891685 1599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:38:17.893816 1599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:17.899863 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.895793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:40055 (local address 127.0.0.1:49324)
0924 13:38:17.895912 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.895949 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.896016 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.896350 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.896354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.896356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:17.896417 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.896419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.897737 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.897738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.898686 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.898688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.899580 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.899581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.899659 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.899671 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.899744 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.899792 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":59,"thread_start_us":51,"threads_started":1}
I20260924 13:38:17.899932 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.896050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.25.240.67:40055)
0924 13:38:17.896206 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.896210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.896220 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:17.896262 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.896263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.896275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.896357 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:17.896526 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.897647 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.898777 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.899401 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.899471 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.899545 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.899777 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.899819 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260924 13:38:17.900317 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.896067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.25.240.65:40055)
0924 13:38:17.896487 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.896492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.896502 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:17.896574 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.896576 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.896593 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.896811 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:17.896873 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.897939 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.899221 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.899974 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.900067 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.900136 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.900219 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.900247 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":88,"threads_started":1}
I20260924 13:38:17.900324 2101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.895799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40055 (local address 127.0.0.1:45530)
0924 13:38:17.896024 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.896035 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.896086 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.896693 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.896694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.896696 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:17.896743 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.896744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.898016 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.898018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.899138 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.899140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.900079 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.900080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.900141 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.900145 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.900209 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.900247 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":66,"threads_started":1}
I20260924 13:38:17.900820 2016 tablet_service.cc:1467] Processing CreateTablet for tablet 94970d1d463e4308b324e7ac1a284597 (DEFAULT_TABLE table=test-table [id=81d00bde2fb049a89d836abbc845eea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:17.900929 1736 tablet_service.cc:1467] Processing CreateTablet for tablet 94970d1d463e4308b324e7ac1a284597 (DEFAULT_TABLE table=test-table [id=81d00bde2fb049a89d836abbc845eea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:17.901181 2016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94970d1d463e4308b324e7ac1a284597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:17.901289 1736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94970d1d463e4308b324e7ac1a284597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:17.904076 2107 tablet_bootstrap.cc:492] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: Bootstrap starting.
I20260924 13:38:17.904309 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.895818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:42093 (local address 127.0.0.1:58834)
0924 13:38:17.896123 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.896135 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.896172 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.896680 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.896681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.896683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:17.896735 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.896737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.897667 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.897668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.899042 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.899045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.904075 (+ 5030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.904077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.904144 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.904149 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.904193 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.904236 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":38,"thread_start_us":51,"threads_started":1}
I20260924 13:38:17.904429 2108 tablet_bootstrap.cc:492] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: Bootstrap starting.
I20260924 13:38:17.904727 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.896289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.25.240.66:42093)
0924 13:38:17.896471 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.896475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.896485 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:17.896521 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.896523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.896539 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.896612 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:17.896795 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.897601 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.899143 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.899831 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.899900 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.904555 (+ 4655us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.904584 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.904634 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260924 13:38:17.904973 2108 tablet_bootstrap.cc:654] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:17.904973 2107 tablet_bootstrap.cc:654] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:17.905313 2108 log.cc:824] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:17.905313 2107 log.cc:824] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:17.905323 1876 tablet_service.cc:1467] Processing CreateTablet for tablet 94970d1d463e4308b324e7ac1a284597 (DEFAULT_TABLE table=test-table [id=81d00bde2fb049a89d836abbc845eea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:17.905644 1876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94970d1d463e4308b324e7ac1a284597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:17.906076 2108 tablet_bootstrap.cc:492] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: No bootstrap required, opened a new log
I20260924 13:38:17.906153 2108 ts_tablet_manager.cc:1397] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:17.906368 2107 tablet_bootstrap.cc:492] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: No bootstrap required, opened a new log
I20260924 13:38:17.906436 2107 ts_tablet_manager.cc:1397] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:17.908144 2108 raft_consensus.cc:348] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.908314 2108 raft_consensus.cc:374] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:17.908358 2108 raft_consensus.cc:729] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0dcffeabdb1472297753670bf8bd10d, State: Initialized, Role: FOLLOWER
I20260924 13:38:17.908329 2107 raft_consensus.cc:348] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.908470 2107 raft_consensus.cc:374] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:17.908509 2107 raft_consensus.cc:729] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2df2750c104dbd9f38deee5be88a99, State: Initialized, Role: FOLLOWER
I20260924 13:38:17.908499 2108 consensus_queue.cc:260] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.908617 2107 consensus_queue.cc:260] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.908838 1802 heartbeater.cc:503] Master 127.25.240.126:46827 was elected leader, sending a full tablet report...
I20260924 13:38:17.908856 2107 ts_tablet_manager.cc:1428] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 13:38:17.908967 2082 heartbeater.cc:503] Master 127.25.240.126:46827 was elected leader, sending a full tablet report...
I20260924 13:38:17.908785 2108 ts_tablet_manager.cc:1428] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:38:17.909108 2112 tablet_bootstrap.cc:492] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Bootstrap starting.
I20260924 13:38:17.909780 2112 tablet_bootstrap.cc:654] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:17.910137 2112 log.cc:824] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:17.910763 2112 tablet_bootstrap.cc:492] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: No bootstrap required, opened a new log
I20260924 13:38:17.910815 2112 ts_tablet_manager.cc:1397] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:17.912128 2112 raft_consensus.cc:348] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.912245 2112 raft_consensus.cc:374] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:17.912300 2112 raft_consensus.cc:729] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ae328ed834eda806797a52459a79f, State: Initialized, Role: FOLLOWER
I20260924 13:38:17.912417 2112 consensus_queue.cc:260] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.912670 2112 ts_tablet_manager.cc:1428] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:17.912721 1942 heartbeater.cc:503] Master 127.25.240.126:46827 was elected leader, sending a full tablet report...
W20260924 13:38:17.944162 1943 tablet.cc:2369] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:38:17.979879 2111 raft_consensus.cc:482] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:17.980003 2111 raft_consensus.cc:504] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.980314 2111 leader_election.cc:288] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055), 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093)
I20260924 13:38:17.984248 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.980632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:40139 (local address 127.25.240.67:40055)
0924 13:38:17.980692 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.980694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.980704 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:17.980733 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.980733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.980743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.980806 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:17.980995 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.981784 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.983308 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.983876 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.983952 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.983961 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.984142 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.984179 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:38:17.984248 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.980608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:32919 (local address 127.25.240.66:42093)
0924 13:38:17.980672 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:17.980674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:17.980866 (+ 192us) server_negotiation.cc:407] Connection header received
0924 13:38:17.980897 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:17.980898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:17.980905 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:17.980947 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:17.981081 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.981798 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.983308 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.983885 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.983952 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.983961 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:17.984143 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 13:38:17.984179 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:38:17.984277 2116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.980514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:42093 (local address 127.25.240.65:32919)
0924 13:38:17.980772 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.980780 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.980852 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.980951 (+ 99us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.980952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.980953 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:17.981009 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.981010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.981869 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.981870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.983220 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.983223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.983962 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.983963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.984061 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.984067 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.984127 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.984165 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":169,"thread_start_us":110,"threads_started":1}
I20260924 13:38:17.984282 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:17.980509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:40055 (local address 127.25.240.65:40139)
0924 13:38:17.980577 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:17.980592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:17.980650 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:17.980839 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:17.980840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:17.980841 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:17.980917 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.980918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.981869 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.981870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.983220 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:17.983223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:17.983962 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:17.983963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:17.984061 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:17.984067 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:38:17.984127 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 13:38:17.984165 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":79}
I20260924 13:38:17.984690 1896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94970d1d463e4308b324e7ac1a284597" candidate_uuid: "f0dcffeabdb1472297753670bf8bd10d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ae328ed834eda806797a52459a79f" is_pre_election: true
I20260924 13:38:17.984879 1896 raft_consensus.cc:2454] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0dcffeabdb1472297753670bf8bd10d in term 0.
I20260924 13:38:17.984886 2035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94970d1d463e4308b324e7ac1a284597" candidate_uuid: "f0dcffeabdb1472297753670bf8bd10d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2df2750c104dbd9f38deee5be88a99" is_pre_election: true
I20260924 13:38:17.985014 2035 raft_consensus.cc:2454] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0dcffeabdb1472297753670bf8bd10d in term 0.
I20260924 13:38:17.985126 1689 leader_election.cc:302] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, f0dcffeabdb1472297753670bf8bd10d; no voters:
I20260924 13:38:17.985257 2111 raft_consensus.cc:2781] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:17.985313 2111 raft_consensus.cc:482] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:17.985337 2111 raft_consensus.cc:3037] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:17.986281 2111 raft_consensus.cc:504] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.986416 2111 leader_election.cc:288] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [CANDIDATE]: Term 1 election: Requested vote from peers bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055), 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093)
I20260924 13:38:17.986546 2035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94970d1d463e4308b324e7ac1a284597" candidate_uuid: "f0dcffeabdb1472297753670bf8bd10d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2df2750c104dbd9f38deee5be88a99"
I20260924 13:38:17.986565 1896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94970d1d463e4308b324e7ac1a284597" candidate_uuid: "f0dcffeabdb1472297753670bf8bd10d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ae328ed834eda806797a52459a79f"
I20260924 13:38:17.986629 2035 raft_consensus.cc:3037] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:17.986627 1896 raft_consensus.cc:3037] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:17.987496 2035 raft_consensus.cc:2454] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0dcffeabdb1472297753670bf8bd10d in term 1.
I20260924 13:38:17.987496 1896 raft_consensus.cc:2454] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0dcffeabdb1472297753670bf8bd10d in term 1.
I20260924 13:38:17.987641 1689 leader_election.cc:302] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, f0dcffeabdb1472297753670bf8bd10d; no voters:
I20260924 13:38:17.987746 2111 raft_consensus.cc:2781] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:17.987913 2111 raft_consensus.cc:686] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 1 LEADER]: Becoming Leader. State: Replica: f0dcffeabdb1472297753670bf8bd10d, State: Running, Role: LEADER
I20260924 13:38:17.988013 2111 consensus_queue.cc:237] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } }
I20260924 13:38:17.988797 1599 catalog_manager.cc:5495] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d reported cstate change: term changed from 0 to 1, leader changed from <none> to f0dcffeabdb1472297753670bf8bd10d (127.25.240.65). New cstate: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } health_report { overall_health: UNKNOWN } } }
I20260924 13:38:18.014803 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.012597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.25.240.65:40055)
0924 13:38:18.012640 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.012642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.012651 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:18.012707 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.012708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.012725 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.012794 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.012893 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.013356 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.014142 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.014324 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.014379 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.014495 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.014595 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.014651 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.014686 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.014703 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:38:18.016645 1896 raft_consensus.cc:1261] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Refusing update from remote peer f0dcffeabdb1472297753670bf8bd10d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.016647 2035 raft_consensus.cc:1261] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Refusing update from remote peer f0dcffeabdb1472297753670bf8bd10d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.017009 2117 consensus_queue.cc:1035] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.017149 2111 consensus_queue.cc:1035] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.020596 2122 mvcc.cc:205] Tried to move back new op lower bound from 7332893073473245184 to 7332893073359405056. Current Snapshot: MvccSnapshot[applied={T|T < 7332893073473245184 or (T in {7332893073473245184})}]
I20260924 13:38:18.021651 1599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:18.021739 1599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:18.022492 1599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: Sending DeleteTablet for 3 replicas of tablet 94970d1d463e4308b324e7ac1a284597
I20260924 13:38:18.022812 2016 tablet_service.cc:1514] Processing DeleteTablet for tablet 94970d1d463e4308b324e7ac1a284597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:49324
I20260924 13:38:18.022816 1736 tablet_service.cc:1514] Processing DeleteTablet for tablet 94970d1d463e4308b324e7ac1a284597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:45530
I20260924 13:38:18.022934 1876 tablet_service.cc:1514] Processing DeleteTablet for tablet 94970d1d463e4308b324e7ac1a284597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:58834
I20260924 13:38:18.023010 2131 tablet_replica.cc:330] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: stopping tablet replica
I20260924 13:38:18.023095 2131 raft_consensus.cc:2229] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:18.023207 2131 raft_consensus.cc:2258] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.023214 1599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:38:18.023356 1599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:18.023628 2130 tablet_replica.cc:330] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: stopping tablet replica
I20260924 13:38:18.023663 2131 ts_tablet_manager.cc:1905] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.024744 1736 tablet_service.cc:1467] Processing CreateTablet for tablet c63df53129124f2093261117ca664b01 (DEFAULT_TABLE table=test-table [id=58179e9980ce4cbcb6a26830ae7e7a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.024744 1876 tablet_service.cc:1467] Processing CreateTablet for tablet c63df53129124f2093261117ca664b01 (DEFAULT_TABLE table=test-table [id=58179e9980ce4cbcb6a26830ae7e7a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.024869 1736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63df53129124f2093261117ca664b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.024871 1876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63df53129124f2093261117ca664b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.025933 2131 ts_tablet_manager.cc:1918] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:18.025995 2131 log.cc:1196] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/94970d1d463e4308b324e7ac1a284597
I20260924 13:38:18.026239 2131 ts_tablet_manager.cc:1939] T 94970d1d463e4308b324e7ac1a284597 P bf2df2750c104dbd9f38deee5be88a99: Deleting consensus metadata
I20260924 13:38:18.026407 2108 tablet_bootstrap.cc:492] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: Bootstrap starting.
I20260924 13:38:18.026430 2112 tablet_bootstrap.cc:492] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: Bootstrap starting.
I20260924 13:38:18.026855 2112 tablet_bootstrap.cc:654] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.026855 2108 tablet_bootstrap.cc:654] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.027558 1584 catalog_manager.cc:4841] TS bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055): tablet 94970d1d463e4308b324e7ac1a284597 (table test-table [id=81d00bde2fb049a89d836abbc845eea0]) successfully deleted
I20260924 13:38:18.027603 2016 tablet_service.cc:1467] Processing CreateTablet for tablet c63df53129124f2093261117ca664b01 (DEFAULT_TABLE table=test-table [id=58179e9980ce4cbcb6a26830ae7e7a40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.027700 2016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63df53129124f2093261117ca664b01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.027873 2112 tablet_bootstrap.cc:492] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: No bootstrap required, opened a new log
I20260924 13:38:18.027921 2112 ts_tablet_manager.cc:1397] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:18.028079 2112 raft_consensus.cc:348] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.028159 2112 raft_consensus.cc:374] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.028187 2112 raft_consensus.cc:729] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ae328ed834eda806797a52459a79f, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.028231 2112 consensus_queue.cc:260] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.028327 2112 ts_tablet_manager.cc:1428] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:18.028571 2132 tablet_replica.cc:330] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: stopping tablet replica
I20260924 13:38:18.028788 2130 raft_consensus.cc:2229] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:38:18.028898 2107 tablet_bootstrap.cc:492] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: Bootstrap starting.
I20260924 13:38:18.029278 2107 tablet_bootstrap.cc:654] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.029095 2108 tablet_bootstrap.cc:492] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: No bootstrap required, opened a new log
I20260924 13:38:18.029647 2132 raft_consensus.cc:2229] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:18.029666 2108 ts_tablet_manager.cc:1397] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:38:18.029748 2130 raft_consensus.cc:2258] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.029770 2132 raft_consensus.cc:2258] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.029822 2108 raft_consensus.cc:348] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.029910 2108 raft_consensus.cc:374] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.029942 2108 raft_consensus.cc:729] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0dcffeabdb1472297753670bf8bd10d, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.030014 2108 consensus_queue.cc:260] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.030131 2108 ts_tablet_manager.cc:1428] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:18.030222 2130 ts_tablet_manager.cc:1905] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.030501 2132 ts_tablet_manager.cc:1905] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.030596 2107 tablet_bootstrap.cc:492] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: No bootstrap required, opened a new log
I20260924 13:38:18.030642 2107 ts_tablet_manager.cc:1397] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:18.030797 2107 raft_consensus.cc:348] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.030865 2107 raft_consensus.cc:374] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.030892 2107 raft_consensus.cc:729] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2df2750c104dbd9f38deee5be88a99, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.030928 2107 consensus_queue.cc:260] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.031018 2107 ts_tablet_manager.cc:1428] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:18.032116 2130 ts_tablet_manager.cc:1918] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:18.032193 2130 log.cc:1196] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/94970d1d463e4308b324e7ac1a284597
I20260924 13:38:18.032322 2132 ts_tablet_manager.cc:1918] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:18.032437 2130 ts_tablet_manager.cc:1939] T 94970d1d463e4308b324e7ac1a284597 P f0dcffeabdb1472297753670bf8bd10d: Deleting consensus metadata
I20260924 13:38:18.032466 2132 log.cc:1196] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/94970d1d463e4308b324e7ac1a284597
I20260924 13:38:18.032828 1583 catalog_manager.cc:4841] TS f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055): tablet 94970d1d463e4308b324e7ac1a284597 (table test-table [id=81d00bde2fb049a89d836abbc845eea0]) successfully deleted
I20260924 13:38:18.032886 2132 ts_tablet_manager.cc:1939] T 94970d1d463e4308b324e7ac1a284597 P 8f1ae328ed834eda806797a52459a79f: Deleting consensus metadata
I20260924 13:38:18.033259 1585 catalog_manager.cc:4841] TS 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093): tablet 94970d1d463e4308b324e7ac1a284597 (table test-table [id=81d00bde2fb049a89d836abbc845eea0]) successfully deleted
W20260924 13:38:18.049499 1803 tablet.cc:2369] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:38:18.106828 2115 raft_consensus.cc:482] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:18.106945 2115 raft_consensus.cc:504] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.107290 2115 leader_election.cc:288] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055), bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055)
W20260924 13:38:18.107800 2083 tablet.cc:2369] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:38:18.110690 1835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.107489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:40055 (local address 127.25.240.66:38615)
0924 13:38:18.107554 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.107567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.107615 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.107804 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.107805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.107807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:18.107877 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.107879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.108626 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.108627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.109670 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.109673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.110415 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.110417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.110494 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.110498 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.110569 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.110618 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 13:38:18.110695 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.107541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:38615 (local address 127.25.240.67:40055)
0924 13:38:18.107583 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.107585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.107597 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:38:18.107701 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.107702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.107712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.107759 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:18.107955 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.108539 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.109764 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.110326 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.110407 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.110416 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.110588 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.110626 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:38:18.110991 2035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63df53129124f2093261117ca664b01" candidate_uuid: "8f1ae328ed834eda806797a52459a79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2df2750c104dbd9f38deee5be88a99" is_pre_election: true
I20260924 13:38:18.111057 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.107867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:45493 (local address 127.25.240.65:40055)
0924 13:38:18.107914 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.107916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.107923 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:18.107949 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.107949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.107959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.108003 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:18.108244 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.108987 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.110122 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.110552 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.110612 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.110619 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.110952 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.110992 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:38:18.111088 2035 raft_consensus.cc:2454] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1ae328ed834eda806797a52459a79f in term 0.
I20260924 13:38:18.111279 2136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.107494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40055 (local address 127.25.240.66:45493)
0924 13:38:18.107646 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.107655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.107699 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.108085 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.108086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.108087 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:18.108164 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.108165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.109088 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.109089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.110036 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.110038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.110649 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.110650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.110716 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.110720 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.111151 (+ 431us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.111199 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":17,"thread_start_us":96,"threads_started":1}
I20260924 13:38:18.111245 1828 leader_election.cc:302] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, bf2df2750c104dbd9f38deee5be88a99; no voters:
I20260924 13:38:18.111420 2115 raft_consensus.cc:2781] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:18.111474 2115 raft_consensus.cc:482] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:18.111500 2115 raft_consensus.cc:3037] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.112216 2115 raft_consensus.cc:504] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.112354 2115 leader_election.cc:288] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [CANDIDATE]: Term 1 election: Requested vote from peers f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055), bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055)
I20260924 13:38:18.112484 1756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63df53129124f2093261117ca664b01" candidate_uuid: "8f1ae328ed834eda806797a52459a79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0dcffeabdb1472297753670bf8bd10d" is_pre_election: true
I20260924 13:38:18.112546 1755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63df53129124f2093261117ca664b01" candidate_uuid: "8f1ae328ed834eda806797a52459a79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0dcffeabdb1472297753670bf8bd10d"
I20260924 13:38:18.112627 1756 raft_consensus.cc:2454] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1ae328ed834eda806797a52459a79f in term 0.
I20260924 13:38:18.112994 2035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63df53129124f2093261117ca664b01" candidate_uuid: "8f1ae328ed834eda806797a52459a79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2df2750c104dbd9f38deee5be88a99"
I20260924 13:38:18.113080 2035 raft_consensus.cc:3037] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.113706 2035 raft_consensus.cc:2454] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1ae328ed834eda806797a52459a79f in term 1.
I20260924 13:38:18.113894 1828 leader_election.cc:302] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, bf2df2750c104dbd9f38deee5be88a99; no voters: f0dcffeabdb1472297753670bf8bd10d
I20260924 13:38:18.114123 2115 raft_consensus.cc:2781] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:18.114264 2115 raft_consensus.cc:686] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 1 LEADER]: Becoming Leader. State: Replica: 8f1ae328ed834eda806797a52459a79f, State: Running, Role: LEADER
I20260924 13:38:18.114348 2115 consensus_queue.cc:237] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.115075 1598 catalog_manager.cc:5495] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f1ae328ed834eda806797a52459a79f (127.25.240.66). New cstate: current_term: 1 leader_uuid: "8f1ae328ed834eda806797a52459a79f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } health_report { overall_health: UNKNOWN } } }
Sep 24 13:38:18 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257097, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.126@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:18.141269 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.134271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46827 (local address 127.0.0.1:45190)
0924 13:38:18.134417 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.134429 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.134497 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.134650 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.134651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.134929 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:18.134989 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.134991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.135918 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.135919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.136499 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.136501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.136947 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.136948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.137022 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.137596 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:18.137601 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:18.138985 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:18.140356 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:18.140363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:18.140365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:18.140562 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:18.140733 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:18.140733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:18.140733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:18.140790 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:18.140989 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:18.140995 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:18.141036 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.141087 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.141173 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260924 13:38:18.141314 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.134342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.25.240.126:46827)
0924 13:38:18.134403 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.134405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.134504 (+ 99us) server_negotiation.cc:407] Connection header received
0924 13:38:18.134549 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.134550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.134562 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.134641 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:18.135084 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.135842 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.136606 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.136867 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.136946 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.139068 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:18.139070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:18.139070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:18.139077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:18.140288 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:18.140626 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:18.140626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:18.140627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:18.140676 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:18.140841 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:18.140842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:18.140842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:18.140938 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:18.140986 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.141117 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.141219 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":32}
I20260924 13:38:18.143347 1598 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:45190:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 13:38:18.143519 1598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:18.144971 1876 tablet_service.cc:1467] Processing CreateTablet for tablet b60ad0e6b3344bb09b0a9d4ac2198e2f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33bc7e769be54ac48b219b19c9ec96e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:38:18.144976 1736 tablet_service.cc:1467] Processing CreateTablet for tablet b60ad0e6b3344bb09b0a9d4ac2198e2f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33bc7e769be54ac48b219b19c9ec96e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:38:18.145028 2016 tablet_service.cc:1467] Processing CreateTablet for tablet b60ad0e6b3344bb09b0a9d4ac2198e2f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=33bc7e769be54ac48b219b19c9ec96e6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 13:38:18.145151 2016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60ad0e6b3344bb09b0a9d4ac2198e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.145152 1736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60ad0e6b3344bb09b0a9d4ac2198e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.145154 1876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60ad0e6b3344bb09b0a9d4ac2198e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.146435 2112 tablet_bootstrap.cc:492] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: Bootstrap starting.
I20260924 13:38:18.146633 2108 tablet_bootstrap.cc:492] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: Bootstrap starting.
I20260924 13:38:18.146637 2107 tablet_bootstrap.cc:492] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: Bootstrap starting.
I20260924 13:38:18.146863 2112 tablet_bootstrap.cc:654] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.146927 2108 tablet_bootstrap.cc:654] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.146965 2107 tablet_bootstrap.cc:654] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.147543 2108 tablet_bootstrap.cc:492] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: No bootstrap required, opened a new log
I20260924 13:38:18.147596 2108 ts_tablet_manager.cc:1397] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:18.147727 2108 raft_consensus.cc:348] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.147810 2108 raft_consensus.cc:374] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.147835 2108 raft_consensus.cc:729] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0dcffeabdb1472297753670bf8bd10d, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.147898 2112 tablet_bootstrap.cc:492] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: No bootstrap required, opened a new log
I20260924 13:38:18.147883 2108 consensus_queue.cc:260] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.147938 2112 ts_tablet_manager.cc:1397] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:18.147984 2108 ts_tablet_manager.cc:1428] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:18.148022 2127 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.148121 2127 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:38:18.148248 2112 raft_consensus.cc:348] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.148325 2112 raft_consensus.cc:374] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.148356 2112 raft_consensus.cc:729] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ae328ed834eda806797a52459a79f, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.148396 2112 consensus_queue.cc:260] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.148473 2107 tablet_bootstrap.cc:492] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: No bootstrap required, opened a new log
I20260924 13:38:18.148500 2112 ts_tablet_manager.cc:1428] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:18.148516 2107 ts_tablet_manager.cc:1397] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:38:18.148538 2115 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.148630 2115 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:38:18.148650 2107 raft_consensus.cc:348] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.148718 2107 raft_consensus.cc:374] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.148743 2107 raft_consensus.cc:729] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2df2750c104dbd9f38deee5be88a99, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.148814 2107 consensus_queue.cc:260] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.148909 2107 ts_tablet_manager.cc:1428] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:18.148996 2113 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.149114 2113 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:38:18.178627 2127 raft_consensus.cc:482] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:18.178725 2127 raft_consensus.cc:504] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.178915 2127 leader_election.cc:288] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093), bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055)
I20260924 13:38:18.179124 2035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ad0e6b3344bb09b0a9d4ac2198e2f" candidate_uuid: "f0dcffeabdb1472297753670bf8bd10d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2df2750c104dbd9f38deee5be88a99" is_pre_election: true
I20260924 13:38:18.179124 1896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ad0e6b3344bb09b0a9d4ac2198e2f" candidate_uuid: "f0dcffeabdb1472297753670bf8bd10d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ae328ed834eda806797a52459a79f" is_pre_election: true
I20260924 13:38:18.179219 2035 raft_consensus.cc:2454] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0dcffeabdb1472297753670bf8bd10d in term 0.
I20260924 13:38:18.179219 1896 raft_consensus.cc:2454] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0dcffeabdb1472297753670bf8bd10d in term 0.
I20260924 13:38:18.179345 1689 leader_election.cc:302] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, f0dcffeabdb1472297753670bf8bd10d; no voters:
I20260924 13:38:18.179454 2127 raft_consensus.cc:2781] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:18.179497 2127 raft_consensus.cc:482] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:18.179512 2127 raft_consensus.cc:3037] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.180161 2127 raft_consensus.cc:504] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.180302 2127 leader_election.cc:288] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [CANDIDATE]: Term 1 election: Requested vote from peers 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093), bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055)
I20260924 13:38:18.180436 1896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ad0e6b3344bb09b0a9d4ac2198e2f" candidate_uuid: "f0dcffeabdb1472297753670bf8bd10d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ae328ed834eda806797a52459a79f"
I20260924 13:38:18.180451 2035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ad0e6b3344bb09b0a9d4ac2198e2f" candidate_uuid: "f0dcffeabdb1472297753670bf8bd10d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2df2750c104dbd9f38deee5be88a99"
I20260924 13:38:18.180510 2035 raft_consensus.cc:3037] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.180510 1896 raft_consensus.cc:3037] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.181213 1896 raft_consensus.cc:2454] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0dcffeabdb1472297753670bf8bd10d in term 1.
I20260924 13:38:18.181214 2035 raft_consensus.cc:2454] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0dcffeabdb1472297753670bf8bd10d in term 1.
I20260924 13:38:18.181396 1689 leader_election.cc:302] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, f0dcffeabdb1472297753670bf8bd10d; no voters:
I20260924 13:38:18.181514 2127 raft_consensus.cc:2781] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:18.181597 2127 raft_consensus.cc:686] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [term 1 LEADER]: Becoming Leader. State: Replica: f0dcffeabdb1472297753670bf8bd10d, State: Running, Role: LEADER
I20260924 13:38:18.181658 2127 consensus_queue.cc:237] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.181913 2111 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: TxnStatusTablet state changed. Reason: New leader f0dcffeabdb1472297753670bf8bd10d. Latest consensus state: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.182013 2111 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:38:18.182173 2149 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 13:38:18.182312 2149 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 13:38:18.182390 1599 catalog_manager.cc:5495] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d reported cstate change: term changed from 0 to 1, leader changed from <none> to f0dcffeabdb1472297753670bf8bd10d (127.25.240.65). New cstate: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } health_report { overall_health: UNKNOWN } } }
I20260924 13:38:18.197109 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.194051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.25.240.65:40055)
0924 13:38:18.194093 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.194095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.194181 (+ 86us) server_negotiation.cc:407] Connection header received
0924 13:38:18.194266 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.194267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.194279 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.194351 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.194513 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.195264 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.196321 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.196625 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.196697 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.196842 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.196877 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.196929 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.196994 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.197021 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:38:18.197129 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.193982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40055 (local address 127.0.0.1:45548)
0924 13:38:18.194117 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.194128 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.194203 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.194361 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.194362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.194363 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:38:18.194412 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.194432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.195350 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.195351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.196244 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.196246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.196696 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.196697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.196748 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.196763 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.196935 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.196941 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.196976 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.197011 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 13:38:18.198146 2035 raft_consensus.cc:1261] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Refusing update from remote peer f0dcffeabdb1472297753670bf8bd10d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.198211 1896 raft_consensus.cc:1261] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Refusing update from remote peer f0dcffeabdb1472297753670bf8bd10d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.198380 2127 consensus_queue.cc:1035] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.198467 2111 consensus_queue.cc:1035] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.198966 2115 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: TxnStatusTablet state changed. Reason: New leader f0dcffeabdb1472297753670bf8bd10d. Latest consensus state: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.199086 2115 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:38:18.199159 2113 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: TxnStatusTablet state changed. Reason: New leader f0dcffeabdb1472297753670bf8bd10d. Latest consensus state: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.199249 2113 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:38:18.199862 2111 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.199944 2111 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:38:18.200033 2121 mvcc.cc:205] Tried to move back new op lower bound from 7332893074217975808 to 7332893074152398848. Current Snapshot: MvccSnapshot[applied={T|T < 7332893074217975808}]
I20260924 13:38:18.200042 2111 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.200089 2111 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P f0dcffeabdb1472297753670bf8bd10d: This TxnStatusTablet replica's current role is: LEADER
I20260924 13:38:18.200364 2113 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.200453 2113 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P bf2df2750c104dbd9f38deee5be88a99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:38:18.200790 2115 tablet_replica.cc:439] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f0dcffeabdb1472297753670bf8bd10d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } } }
I20260924 13:38:18.200891 2115 tablet_replica.cc:442] T b60ad0e6b3344bb09b0a9d4ac2198e2f P 8f1ae328ed834eda806797a52459a79f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 13:38:18.204882 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.202317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.25.240.66:42093)
0924 13:38:18.202353 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.202354 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.202379 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:38:18.202447 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.202448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.202466 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.202536 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.202613 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.203332 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.204108 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.204418 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.204485 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.204595 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.204715 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.204779 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.204800 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.204819 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:18.209673 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.207210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:36885 (local address 127.25.240.65:40055)
0924 13:38:18.207251 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.207253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.207376 (+ 123us) server_negotiation.cc:407] Connection header received
0924 13:38:18.207431 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.207432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.207441 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.207481 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.207584 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.207997 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.209038 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.209206 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.209253 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.209434 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.209458 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.209494 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.209548 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.209604 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:38:18.209707 2155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.207124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40055 (local address 127.25.240.66:36885)
0924 13:38:18.207304 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.207314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.207375 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.207491 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.207492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.207495 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:38:18.207532 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.207533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.208063 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.208063 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.208953 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.208955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.209289 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.209290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.209342 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.209356 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.209495 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.209496 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.209545 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.209623 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":85,"threads_started":1}
I20260924 13:38:18.211812 2035 raft_consensus.cc:1261] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Refusing update from remote peer 8f1ae328ed834eda806797a52459a79f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.211939 1755 raft_consensus.cc:3037] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.211999 2137 consensus_queue.cc:1035] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.212896 1755 raft_consensus.cc:1261] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Refusing update from remote peer 8f1ae328ed834eda806797a52459a79f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.213177 2137 consensus_queue.cc:1035] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.214556 2120 mvcc.cc:205] Tried to move back new op lower bound from 7332893074274140160 to 7332893073876729856. Current Snapshot: MvccSnapshot[applied={T|T < 7332893074274140160}]
I20260924 13:38:18.220139 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.217621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:43157 (local address 127.25.240.66:42093)
0924 13:38:18.217658 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.217660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.217799 (+ 139us) server_negotiation.cc:407] Connection header received
0924 13:38:18.217871 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.217872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.217883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.217937 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.218030 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.218479 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.219418 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.219675 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.219709 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.219917 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.219950 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.219997 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.220061 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.220079 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:38:18.220176 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.217583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:42093 (local address 127.25.240.65:43157)
0924 13:38:18.217722 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.217733 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.217814 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.217940 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.217941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.217942 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:38:18.217984 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.217985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.218565 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.218566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.219316 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.219319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.219755 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.219755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.219838 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.219853 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.219996 (+ 143us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.219997 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.220056 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.220097 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260924 13:38:18.229665 1598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:18.229765 1598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:18.230404 1598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: Sending DeleteTablet for 3 replicas of tablet c63df53129124f2093261117ca664b01
I20260924 13:38:18.230686 1876 tablet_service.cc:1514] Processing DeleteTablet for tablet c63df53129124f2093261117ca664b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:58834
I20260924 13:38:18.230733 1736 tablet_service.cc:1514] Processing DeleteTablet for tablet c63df53129124f2093261117ca664b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:45530
I20260924 13:38:18.230791 2132 tablet_replica.cc:330] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: stopping tablet replica
I20260924 13:38:18.230827 2130 tablet_replica.cc:330] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: stopping tablet replica
I20260924 13:38:18.230849 2132 raft_consensus.cc:2229] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:38:18.230887 2130 raft_consensus.cc:2229] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:18.230914 2132 raft_consensus.cc:2258] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.230930 2130 raft_consensus.cc:2258] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.231196 2016 tablet_service.cc:1514] Processing DeleteTablet for tablet c63df53129124f2093261117ca664b01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:49324
I20260924 13:38:18.231231 2132 ts_tablet_manager.cc:1905] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.231232 2130 ts_tablet_manager.cc:1905] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.231280 2131 tablet_replica.cc:330] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: stopping tablet replica
I20260924 13:38:18.231320 2131 raft_consensus.cc:2229] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:18.231350 2131 raft_consensus.cc:2258] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.231637 2131 ts_tablet_manager.cc:1905] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.232838 2130 ts_tablet_manager.cc:1918] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:38:18.232872 2132 ts_tablet_manager.cc:1918] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:38:18.232892 2130 log.cc:1196] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/c63df53129124f2093261117ca664b01
I20260924 13:38:18.232926 2132 log.cc:1196] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/c63df53129124f2093261117ca664b01
I20260924 13:38:18.233121 2130 ts_tablet_manager.cc:1939] T c63df53129124f2093261117ca664b01 P f0dcffeabdb1472297753670bf8bd10d: Deleting consensus metadata
I20260924 13:38:18.233141 2132 ts_tablet_manager.cc:1939] T c63df53129124f2093261117ca664b01 P 8f1ae328ed834eda806797a52459a79f: Deleting consensus metadata
I20260924 13:38:18.233784 1585 catalog_manager.cc:4841] TS 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093): tablet c63df53129124f2093261117ca664b01 (table test-table [id=58179e9980ce4cbcb6a26830ae7e7a40]) successfully deleted
I20260924 13:38:18.233807 1583 catalog_manager.cc:4841] TS f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055): tablet c63df53129124f2093261117ca664b01 (table test-table [id=58179e9980ce4cbcb6a26830ae7e7a40]) successfully deleted
I20260924 13:38:18.237093 2131 ts_tablet_manager.cc:1918] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:38:18.237152 2131 log.cc:1196] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/c63df53129124f2093261117ca664b01
I20260924 13:38:18.237360 2131 ts_tablet_manager.cc:1939] T c63df53129124f2093261117ca664b01 P bf2df2750c104dbd9f38deee5be88a99: Deleting consensus metadata
I20260924 13:38:18.237709 1584 catalog_manager.cc:4841] TS bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055): tablet c63df53129124f2093261117ca664b01 (table test-table [id=58179e9980ce4cbcb6a26830ae7e7a40]) successfully deleted
Sep 24 13:38:18 dist-test-slave-m5z1 krb5kdc[1554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 13:38:18 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:18 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:38:18.260401 1648 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:18.254820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.25.240.126:46827)
0924 13:38:18.254870 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.254873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.255011 (+ 138us) server_negotiation.cc:407] Connection header received
0924 13:38:18.255095 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.255096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.255107 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.255176 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:18.255736 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.256240 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.256976 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.257185 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.257263 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.260331 (+ 3068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45192: BlockingRecv error: recv got EOF from 127.0.0.1:45192 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 13:38:18 dist-test-slave-m5z1 krb5kdc[1554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM
I20260924 13:38:18.268587 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.261448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.25.240.126:46827)
0924 13:38:18.261487 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.261489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.261738 (+ 249us) server_negotiation.cc:407] Connection header received
0924 13:38:18.261807 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.261808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.261818 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.261869 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:18.262331 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.262870 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.263565 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.263759 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.263844 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.266060 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:18.266061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:18.266062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:18.266069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:18.267449 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:18.267805 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:18.267806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:18.267807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:18.267867 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:18.268070 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:18.268071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:18.268071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:18.268197 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:18.268305 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.268403 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.268478 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:38:18.270249 1598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:38:18.270438 1598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:18.271713 1736 tablet_service.cc:1467] Processing CreateTablet for tablet 542d7639b61a4c67a05e821582b7f6c3 (DEFAULT_TABLE table=test-table [id=3bdf0166f3714a93b1917acc5dc2c163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.271849 1736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542d7639b61a4c67a05e821582b7f6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.271834 1876 tablet_service.cc:1467] Processing CreateTablet for tablet 542d7639b61a4c67a05e821582b7f6c3 (DEFAULT_TABLE table=test-table [id=3bdf0166f3714a93b1917acc5dc2c163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.271946 1876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542d7639b61a4c67a05e821582b7f6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.273197 2108 tablet_bootstrap.cc:492] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: Bootstrap starting.
I20260924 13:38:18.273250 2016 tablet_service.cc:1467] Processing CreateTablet for tablet 542d7639b61a4c67a05e821582b7f6c3 (DEFAULT_TABLE table=test-table [id=3bdf0166f3714a93b1917acc5dc2c163]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.273356 2016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542d7639b61a4c67a05e821582b7f6c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.273510 2108 tablet_bootstrap.cc:654] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.274629 2107 tablet_bootstrap.cc:492] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: Bootstrap starting.
I20260924 13:38:18.274662 2112 tablet_bootstrap.cc:492] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: Bootstrap starting.
I20260924 13:38:18.275089 2112 tablet_bootstrap.cc:654] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.275087 2107 tablet_bootstrap.cc:654] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.276001 2107 tablet_bootstrap.cc:492] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: No bootstrap required, opened a new log
I20260924 13:38:18.276051 2107 ts_tablet_manager.cc:1397] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:18.276166 2108 tablet_bootstrap.cc:492] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: No bootstrap required, opened a new log
I20260924 13:38:18.276206 2108 ts_tablet_manager.cc:1397] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:38:18.276213 2107 raft_consensus.cc:348] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.276280 2107 raft_consensus.cc:374] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.276300 2107 raft_consensus.cc:729] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2df2750c104dbd9f38deee5be88a99, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.276300 2108 raft_consensus.cc:348] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.276361 2108 raft_consensus.cc:374] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.276345 2107 consensus_queue.cc:260] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.276381 2108 raft_consensus.cc:729] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0dcffeabdb1472297753670bf8bd10d, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.276419 2108 consensus_queue.cc:260] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.276449 2107 ts_tablet_manager.cc:1428] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:18.276510 2108 ts_tablet_manager.cc:1428] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:18.278020 2112 tablet_bootstrap.cc:492] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: No bootstrap required, opened a new log
I20260924 13:38:18.278096 2112 ts_tablet_manager.cc:1397] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:38:18.278251 2112 raft_consensus.cc:348] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.278329 2112 raft_consensus.cc:374] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.278358 2112 raft_consensus.cc:729] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ae328ed834eda806797a52459a79f, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.278398 2112 consensus_queue.cc:260] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.278498 2112 ts_tablet_manager.cc:1428] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:18.412923 2161 raft_consensus.cc:482] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:18.413021 2161 raft_consensus.cc:504] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.413245 2161 leader_election.cc:288] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055), bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055)
I20260924 13:38:18.413491 1755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542d7639b61a4c67a05e821582b7f6c3" candidate_uuid: "8f1ae328ed834eda806797a52459a79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0dcffeabdb1472297753670bf8bd10d" is_pre_election: true
I20260924 13:38:18.413470 2035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542d7639b61a4c67a05e821582b7f6c3" candidate_uuid: "8f1ae328ed834eda806797a52459a79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2df2750c104dbd9f38deee5be88a99" is_pre_election: true
I20260924 13:38:18.413623 1755 raft_consensus.cc:2454] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1ae328ed834eda806797a52459a79f in term 0.
I20260924 13:38:18.413626 2035 raft_consensus.cc:2454] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1ae328ed834eda806797a52459a79f in term 0.
I20260924 13:38:18.413803 1827 leader_election.cc:302] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, f0dcffeabdb1472297753670bf8bd10d; no voters:
I20260924 13:38:18.413918 2161 raft_consensus.cc:2781] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:18.413956 2161 raft_consensus.cc:482] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:18.413976 2161 raft_consensus.cc:3037] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.414721 2161 raft_consensus.cc:504] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.414865 2161 leader_election.cc:288] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [CANDIDATE]: Term 1 election: Requested vote from peers f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055), bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055)
I20260924 13:38:18.415041 2035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542d7639b61a4c67a05e821582b7f6c3" candidate_uuid: "8f1ae328ed834eda806797a52459a79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2df2750c104dbd9f38deee5be88a99"
I20260924 13:38:18.415114 2035 raft_consensus.cc:3037] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.415042 1755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542d7639b61a4c67a05e821582b7f6c3" candidate_uuid: "8f1ae328ed834eda806797a52459a79f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0dcffeabdb1472297753670bf8bd10d"
I20260924 13:38:18.415140 1755 raft_consensus.cc:3037] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.415695 2035 raft_consensus.cc:2454] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1ae328ed834eda806797a52459a79f in term 1.
I20260924 13:38:18.415820 1755 raft_consensus.cc:2454] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1ae328ed834eda806797a52459a79f in term 1.
I20260924 13:38:18.415867 1828 leader_election.cc:302] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, bf2df2750c104dbd9f38deee5be88a99; no voters:
I20260924 13:38:18.416014 2161 raft_consensus.cc:2781] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:18.416071 2161 raft_consensus.cc:686] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 1 LEADER]: Becoming Leader. State: Replica: 8f1ae328ed834eda806797a52459a79f, State: Running, Role: LEADER
I20260924 13:38:18.416126 2161 consensus_queue.cc:237] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.416688 1598 catalog_manager.cc:5495] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f1ae328ed834eda806797a52459a79f (127.25.240.66). New cstate: current_term: 1 leader_uuid: "8f1ae328ed834eda806797a52459a79f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } health_report { overall_health: UNKNOWN } } }
I20260924 13:38:18.423321 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.421370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.25.240.66:42093)
0924 13:38:18.421434 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.421437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.421444 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:18.421464 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.421464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.421472 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.421518 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.421680 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.422134 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.422824 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.422994 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.423050 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.423156 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.423179 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.423211 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.423258 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.423272 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:38:18.424131 2035 raft_consensus.cc:1261] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Refusing update from remote peer 8f1ae328ed834eda806797a52459a79f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.424134 1755 raft_consensus.cc:1261] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Refusing update from remote peer 8f1ae328ed834eda806797a52459a79f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.424356 2161 consensus_queue.cc:1035] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.424444 2115 consensus_queue.cc:1035] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.426594 1598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:18.426680 1598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:18.427166 1598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: Sending DeleteTablet for 3 replicas of tablet 542d7639b61a4c67a05e821582b7f6c3
I20260924 13:38:18.427423 1876 tablet_service.cc:1514] Processing DeleteTablet for tablet 542d7639b61a4c67a05e821582b7f6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:58834
I20260924 13:38:18.427428 1736 tablet_service.cc:1514] Processing DeleteTablet for tablet 542d7639b61a4c67a05e821582b7f6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:45530
I20260924 13:38:18.427528 2130 tablet_replica.cc:330] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: stopping tablet replica
I20260924 13:38:18.427546 2016 tablet_service.cc:1514] Processing DeleteTablet for tablet 542d7639b61a4c67a05e821582b7f6c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:49324
I20260924 13:38:18.427556 2132 tablet_replica.cc:330] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: stopping tablet replica
I20260924 13:38:18.427574 2130 raft_consensus.cc:2229] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:18.427613 2130 raft_consensus.cc:2258] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.427616 2132 raft_consensus.cc:2229] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:38:18.427631 2131 tablet_replica.cc:330] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: stopping tablet replica
I20260924 13:38:18.427662 2132 raft_consensus.cc:2258] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.427681 2131 raft_consensus.cc:2229] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:18.427728 2131 raft_consensus.cc:2258] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.427955 2130 ts_tablet_manager.cc:1905] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.427963 2132 ts_tablet_manager.cc:1905] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.427935 1598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:38:18.428066 1598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:18.428222 2131 ts_tablet_manager.cc:1905] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.429097 1876 tablet_service.cc:1467] Processing CreateTablet for tablet a47131658ebd4acb87bd844b9cfa3560 (DEFAULT_TABLE table=test-table [id=ba9668ce98ca4ba1b7d0402e3319790a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.429191 1876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47131658ebd4acb87bd844b9cfa3560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.429659 2132 ts_tablet_manager.cc:1918] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:18.429693 2130 ts_tablet_manager.cc:1918] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:18.429711 2132 log.cc:1196] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/542d7639b61a4c67a05e821582b7f6c3
I20260924 13:38:18.429749 2130 log.cc:1196] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/542d7639b61a4c67a05e821582b7f6c3
I20260924 13:38:18.429931 2132 ts_tablet_manager.cc:1939] T 542d7639b61a4c67a05e821582b7f6c3 P 8f1ae328ed834eda806797a52459a79f: Deleting consensus metadata
I20260924 13:38:18.429927 2131 ts_tablet_manager.cc:1918] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:18.429976 2131 log.cc:1196] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/542d7639b61a4c67a05e821582b7f6c3
I20260924 13:38:18.429972 2130 ts_tablet_manager.cc:1939] T 542d7639b61a4c67a05e821582b7f6c3 P f0dcffeabdb1472297753670bf8bd10d: Deleting consensus metadata
I20260924 13:38:18.430238 2131 ts_tablet_manager.cc:1939] T 542d7639b61a4c67a05e821582b7f6c3 P bf2df2750c104dbd9f38deee5be88a99: Deleting consensus metadata
I20260924 13:38:18.430310 1585 catalog_manager.cc:4841] TS 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093): tablet 542d7639b61a4c67a05e821582b7f6c3 (table test-table [id=3bdf0166f3714a93b1917acc5dc2c163]) successfully deleted
I20260924 13:38:18.430531 2112 tablet_bootstrap.cc:492] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: Bootstrap starting.
I20260924 13:38:18.430666 1584 catalog_manager.cc:4841] TS bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055): tablet 542d7639b61a4c67a05e821582b7f6c3 (table test-table [id=3bdf0166f3714a93b1917acc5dc2c163]) successfully deleted
I20260924 13:38:18.430735 2016 tablet_service.cc:1467] Processing CreateTablet for tablet a47131658ebd4acb87bd844b9cfa3560 (DEFAULT_TABLE table=test-table [id=ba9668ce98ca4ba1b7d0402e3319790a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.430819 1583 catalog_manager.cc:4841] TS f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055): tablet 542d7639b61a4c67a05e821582b7f6c3 (table test-table [id=3bdf0166f3714a93b1917acc5dc2c163]) successfully deleted
I20260924 13:38:18.430850 2016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47131658ebd4acb87bd844b9cfa3560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.430881 2112 tablet_bootstrap.cc:654] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.430966 1736 tablet_service.cc:1467] Processing CreateTablet for tablet a47131658ebd4acb87bd844b9cfa3560 (DEFAULT_TABLE table=test-table [id=ba9668ce98ca4ba1b7d0402e3319790a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:18.431068 1736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47131658ebd4acb87bd844b9cfa3560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:18.431464 2112 tablet_bootstrap.cc:492] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: No bootstrap required, opened a new log
I20260924 13:38:18.431514 2112 ts_tablet_manager.cc:1397] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:18.431636 2112 raft_consensus.cc:348] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.431708 2112 raft_consensus.cc:374] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.431733 2112 raft_consensus.cc:729] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ae328ed834eda806797a52459a79f, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.431774 2112 consensus_queue.cc:260] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.431905 2112 ts_tablet_manager.cc:1428] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:18.432037 2107 tablet_bootstrap.cc:492] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: Bootstrap starting.
I20260924 13:38:18.432381 2108 tablet_bootstrap.cc:492] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d: Bootstrap starting.
I20260924 13:38:18.432431 2107 tablet_bootstrap.cc:654] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.432717 2108 tablet_bootstrap.cc:654] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:18.433234 2108 tablet_bootstrap.cc:492] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d: No bootstrap required, opened a new log
I20260924 13:38:18.433300 2108 ts_tablet_manager.cc:1397] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:18.433369 2107 tablet_bootstrap.cc:492] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: No bootstrap required, opened a new log
I20260924 13:38:18.433421 2107 ts_tablet_manager.cc:1397] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:18.433398 2108 raft_consensus.cc:348] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.433466 2108 raft_consensus.cc:374] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.433499 2108 raft_consensus.cc:729] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0dcffeabdb1472297753670bf8bd10d, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.433575 2108 consensus_queue.cc:260] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.433602 2107 raft_consensus.cc:348] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.433669 2108 ts_tablet_manager.cc:1428] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:18.433676 2107 raft_consensus.cc:374] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:18.433703 2107 raft_consensus.cc:729] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2df2750c104dbd9f38deee5be88a99, State: Initialized, Role: FOLLOWER
I20260924 13:38:18.433749 2107 consensus_queue.cc:260] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.433842 2107 ts_tablet_manager.cc:1428] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260924 13:38:18.550973 1797 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.25.240.65:40055
I20260924 13:38:18.586104 2113 raft_consensus.cc:482] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:18.586200 2113 raft_consensus.cc:504] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.586496 2113 leader_election.cc:288] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055), 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093)
I20260924 13:38:18.590495 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.586821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:43009 (local address 127.25.240.65:40055)
0924 13:38:18.586865 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.586866 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.586885 (+ 19us) server_negotiation.cc:407] Connection header received
0924 13:38:18.586970 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.586971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.586980 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.587050 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:18.587206 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.587943 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.589537 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.590114 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.590185 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.590194 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.590385 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.590424 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:38:18.590495 2106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.586768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:37457 (local address 127.25.240.66:42093)
0924 13:38:18.586819 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.586821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.586912 (+ 91us) server_negotiation.cc:407] Connection header received
0924 13:38:18.586958 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.586959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.586969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.587043 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:18.587206 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.587958 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.589541 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.590106 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.590185 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.590194 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.590385 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.590424 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 13:38:18.590552 2185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.586697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:42093 (local address 127.25.240.67:37457)
0924 13:38:18.586833 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.586846 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.586905 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.587052 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.587053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.587056 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:18.587138 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.587139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.588032 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.588033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.589449 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.589451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.590221 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.590222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.590304 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.590312 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.590370 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.590432 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":73,"thread_start_us":73,"threads_started":1}
I20260924 13:38:18.590552 2186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.586716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40055 (local address 127.25.240.67:43009)
0924 13:38:18.586834 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.586846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.586901 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.587052 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.587053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.587058 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:18.587132 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.587134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.588060 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.588061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.589449 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.589451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.590221 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.590222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.590304 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.590310 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.590370 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.590432 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":87,"thread_start_us":57,"threads_started":1}
I20260924 13:38:18.590864 1896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47131658ebd4acb87bd844b9cfa3560" candidate_uuid: "bf2df2750c104dbd9f38deee5be88a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ae328ed834eda806797a52459a79f" is_pre_election: true
I20260924 13:38:18.590919 1755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47131658ebd4acb87bd844b9cfa3560" candidate_uuid: "bf2df2750c104dbd9f38deee5be88a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0dcffeabdb1472297753670bf8bd10d" is_pre_election: true
I20260924 13:38:18.590966 1896 raft_consensus.cc:2454] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf2df2750c104dbd9f38deee5be88a99 in term 0.
I20260924 13:38:18.590996 1755 raft_consensus.cc:2454] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf2df2750c104dbd9f38deee5be88a99 in term 0.
I20260924 13:38:18.591126 1969 leader_election.cc:302] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, bf2df2750c104dbd9f38deee5be88a99; no voters:
I20260924 13:38:18.591265 2113 raft_consensus.cc:2781] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:18.591320 2113 raft_consensus.cc:482] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:18.591334 2113 raft_consensus.cc:3037] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.592015 2113 raft_consensus.cc:504] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.592146 2113 leader_election.cc:288] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [CANDIDATE]: Term 1 election: Requested vote from peers f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055), 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093)
I20260924 13:38:18.592290 1896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47131658ebd4acb87bd844b9cfa3560" candidate_uuid: "bf2df2750c104dbd9f38deee5be88a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ae328ed834eda806797a52459a79f"
I20260924 13:38:18.592366 1896 raft_consensus.cc:3037] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.592289 1755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47131658ebd4acb87bd844b9cfa3560" candidate_uuid: "bf2df2750c104dbd9f38deee5be88a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0dcffeabdb1472297753670bf8bd10d"
I20260924 13:38:18.592396 1755 raft_consensus.cc:3037] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:18.592947 1896 raft_consensus.cc:2454] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf2df2750c104dbd9f38deee5be88a99 in term 1.
I20260924 13:38:18.593101 1969 leader_election.cc:302] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ae328ed834eda806797a52459a79f, bf2df2750c104dbd9f38deee5be88a99; no voters:
I20260924 13:38:18.593134 1755 raft_consensus.cc:2454] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf2df2750c104dbd9f38deee5be88a99 in term 1.
I20260924 13:38:18.593194 2113 raft_consensus.cc:2781] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:18.593288 2113 raft_consensus.cc:686] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 1 LEADER]: Becoming Leader. State: Replica: bf2df2750c104dbd9f38deee5be88a99, State: Running, Role: LEADER
I20260924 13:38:18.593344 2113 consensus_queue.cc:237] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } }
I20260924 13:38:18.594072 1598 catalog_manager.cc:5495] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67). New cstate: current_term: 1 leader_uuid: "bf2df2750c104dbd9f38deee5be88a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf2df2750c104dbd9f38deee5be88a99" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 40055 } health_report { overall_health: HEALTHY } } }
I20260924 13:38:18.618531 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.616395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49326 (local address 127.25.240.67:40055)
0924 13:38:18.616492 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.616493 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.616499 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:38:18.616527 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.616528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.616545 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.616609 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.616733 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.617208 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.617920 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.618108 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.618161 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.618271 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.618368 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.618418 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.618448 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.618462 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":61,"threads_started":1}
I20260924 13:38:18.622790 2105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.620835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:59933 (local address 127.25.240.65:40055)
0924 13:38:18.620874 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.620875 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.620924 (+ 49us) server_negotiation.cc:407] Connection header received
0924 13:38:18.620973 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.620974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.620982 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.621027 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.621155 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.621602 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.622272 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.622440 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.622487 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.622618 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.622641 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.622678 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.622726 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.622741 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:18.622818 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.620715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40055 (local address 127.25.240.67:59933)
0924 13:38:18.620872 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.620884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.620935 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.621029 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.621029 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.621030 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:38:18.621107 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.621108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.621662 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.621662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.622207 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.622208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.622498 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.622499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.622545 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.622556 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.622689 (+ 133us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.622691 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.622724 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.622756 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260924 13:38:18.624847 1896 raft_consensus.cc:1261] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Refusing update from remote peer bf2df2750c104dbd9f38deee5be88a99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.624913 1755 raft_consensus.cc:1261] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Refusing update from remote peer bf2df2750c104dbd9f38deee5be88a99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:18.625068 2113 consensus_queue.cc:1035] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1ae328ed834eda806797a52459a79f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 42093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.625180 2187 consensus_queue.cc:1035] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0dcffeabdb1472297753670bf8bd10d" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:18.632113 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.629868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:41923 (local address 127.25.240.67:40055)
0924 13:38:18.629919 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:18.629921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:18.629927 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:38:18.629947 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:18.629947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:18.629956 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:18.630042 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:18.630134 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.630595 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.631395 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.631703 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.631765 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.631927 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.631957 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.632008 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:18.632054 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 13:38:18.632069 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:38:18.632120 2159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.629694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:40055 (local address 127.25.240.65:41923)
0924 13:38:18.629749 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.629760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:18.629795 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:18.630048 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:18.630049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:18.630050 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 13:38:18.630086 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.630087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.630676 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.630676 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.631288 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:18.631290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:18.631774 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:18.631775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:18.631833 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:18.631845 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 13:38:18.632010 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 13:38:18.632011 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 13:38:18.632042 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 13:38:18.632074 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:38:18.638437 1597 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:18.638546 1597 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:18.639210 1597 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d2c22ee5eb341208c05ede4fefe1c30: Sending DeleteTablet for 3 replicas of tablet a47131658ebd4acb87bd844b9cfa3560
I20260924 13:38:18.639477 2016 tablet_service.cc:1514] Processing DeleteTablet for tablet a47131658ebd4acb87bd844b9cfa3560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:49324
I20260924 13:38:18.639576 2131 tablet_replica.cc:330] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: stopping tablet replica
I20260924 13:38:18.639585 1876 tablet_service.cc:1514] Processing DeleteTablet for tablet a47131658ebd4acb87bd844b9cfa3560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:58834
I20260924 13:38:18.639640 2131 raft_consensus.cc:2229] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:38:18.639675 2132 tablet_replica.cc:330] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: stopping tablet replica
I20260924 13:38:18.639668 1736 tablet_service.cc:1514] Processing DeleteTablet for tablet a47131658ebd4acb87bd844b9cfa3560 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:18 UTC) from {username='oryx', principal='oryx/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:45530
I20260924 13:38:18.639720 2131 raft_consensus.cc:2258] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.639736 2132 raft_consensus.cc:2229] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:18.639743 2130 tablet_replica.cc:330] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d: stopping tablet replica
I20260924 13:38:18.639770 2132 raft_consensus.cc:2258] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.639794 2130 raft_consensus.cc:2229] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:18.639834 2130 raft_consensus.cc:2258] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:18.640051 2131 ts_tablet_manager.cc:1905] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.640056 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 1671
I20260924 13:38:18.640102 2130 ts_tablet_manager.cc:1905] T a47131658ebd4acb87bd844b9cfa3560 P f0dcffeabdb1472297753670bf8bd10d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.641671 2131 ts_tablet_manager.cc:1918] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:38:18.641746 2131 log.cc:1196] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/a47131658ebd4acb87bd844b9cfa3560
I20260924 13:38:18.641955 2131 ts_tablet_manager.cc:1939] T a47131658ebd4acb87bd844b9cfa3560 P bf2df2750c104dbd9f38deee5be88a99: Deleting consensus metadata
I20260924 13:38:18.642257 1584 catalog_manager.cc:4841] TS bf2df2750c104dbd9f38deee5be88a99 (127.25.240.67:40055): tablet a47131658ebd4acb87bd844b9cfa3560 (table test-table [id=ba9668ce98ca4ba1b7d0402e3319790a]) successfully deleted
I20260924 13:38:18.645797 2132 ts_tablet_manager.cc:1905] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:18.647040 2132 ts_tablet_manager.cc:1918] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 13:38:18.647113 2132 log.cc:1196] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/a47131658ebd4acb87bd844b9cfa3560
I20260924 13:38:18.647310 2132 ts_tablet_manager.cc:1939] T a47131658ebd4acb87bd844b9cfa3560 P 8f1ae328ed834eda806797a52459a79f: Deleting consensus metadata
I20260924 13:38:18.647627 1585 catalog_manager.cc:4841] TS 8f1ae328ed834eda806797a52459a79f (127.25.240.66:42093): tablet a47131658ebd4acb87bd844b9cfa3560 (table test-table [id=ba9668ce98ca4ba1b7d0402e3319790a]) successfully deleted
W20260924 13:38:18.649946 1583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:40055 (error 108)
I20260924 13:38:18.650197 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 1811
I20260924 13:38:18.650854 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:18.650267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40055 (local address 127.0.0.1:45550)
0924 13:38:18.650776 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:18.650815 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:40055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":37,"threads_started":1}
W20260924 13:38:18.650959 1583 catalog_manager.cc:4584] TS f0dcffeabdb1472297753670bf8bd10d (127.25.240.65:40055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a47131658ebd4acb87bd844b9cfa3560: Network error: Client connection negotiation failed: client connection to 127.25.240.65:40055: connect: Connection refused (error 111)
I20260924 13:38:18.657804 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 1951
I20260924 13:38:18.662925 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 1567
2026-09-24T13:38:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2594 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:18 dist-test-slave-m5z1 krb5kdc[2201](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:18 dist-test-slave-m5z1 krb5kdc[2201](info): set up 2 sockets
Sep 24 13:38:18 dist-test-slave-m5z1 krb5kdc[2201](info): commencing operation
krb5kdc: starting...
W20260924 13:38:19.702302 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:19 dist-test-slave-m5z1 krb5kdc[2201](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:19.817999 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37243
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:37243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:19.897519 2214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:19.897734 2214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:19.897756 2214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:19.898892 2214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:19.898934 2214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:19.898949 2214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:19.898990 2214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:19.899016 2214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:19.900439 2214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:37243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:19.900660 2214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:19.900870 2214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:19.905804 2219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:19.905779 2222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:19.905967 2214 server_base.cc:1034] running on GCE node
W20260924 13:38:19.905778 2220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:19.906320 2214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:19.906685 2214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:19.907860 2214 hybrid_clock.cc:648] HybridClock initialized: now 1790257099907845 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T13:38:19Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1249 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:19 dist-test-slave-m5z1 krb5kdc[2227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:19 dist-test-slave-m5z1 krb5kdc[2227](info): set up 2 sockets
Sep 24 13:38:19 dist-test-slave-m5z1 krb5kdc[2227](info): commencing operation
krb5kdc: starting...
W20260924 13:38:20.970494 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:21.088317 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:40193
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:40193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:21.169032 2240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:21.169237 2240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:21.169258 2240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:21.170446 2240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:21.170491 2240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:21.170506 2240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:21.170519 2240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:21.170560 2240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:21.172016 2240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:40193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:40193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:21.172263 2240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:21.172505 2240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:21.177350 2248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:21.177374 2245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.177507 2240 server_base.cc:1034] running on GCE node
W20260924 13:38:21.177379 2246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.177861 2240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:21.178164 2240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:21.179337 2240 hybrid_clock.cc:648] HybridClock initialized: now 1790257101179332 us; error 36 us; skew 500 ppm
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:21.181842 2240 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:21.182271 2240 webserver.cc:460] Webserver started at http://127.25.240.126:41791/ using document root <none> and password file <none>
I20260924 13:38:21.182511 2240 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:21.182588 2240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:21.182737 2240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:21.183786 2240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "6f8965302e42491daa2fa5be078bbaac"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "65bbc7e34e8d0a781a6c04b683562e64"
server_key_iv: "a6f5e492a1e2bbf4ff6053d071282ae7"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.184149 2240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "6f8965302e42491daa2fa5be078bbaac"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "65bbc7e34e8d0a781a6c04b683562e64"
server_key_iv: "a6f5e492a1e2bbf4ff6053d071282ae7"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.185454 2240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:21.186216 2255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.186378 2240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:21.186476 2240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "6f8965302e42491daa2fa5be078bbaac"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "65bbc7e34e8d0a781a6c04b683562e64"
server_key_iv: "a6f5e492a1e2bbf4ff6053d071282ae7"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.186559 2240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:21.197641 2240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:21.199585 2240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:21.199752 2240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:21.203715 2240 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:40193
I20260924 13:38:21.203826 2307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:40193 every 8 connection(s)
I20260924 13:38:21.204134 2240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:21.205014 2308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:21.207345 2308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Bootstrap starting.
I20260924 13:38:21.207937 2308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:21.208238 2308 log.cc:824] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:21.208978 2308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: No bootstrap required, opened a new log
I20260924 13:38:21.210685 2308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } }
I20260924 13:38:21.210824 2308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:21.210879 2308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8965302e42491daa2fa5be078bbaac, State: Initialized, Role: FOLLOWER
I20260924 13:38:21.211030 2308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } }
I20260924 13:38:21.211090 2308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:21.211131 2308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:21.211184 2308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:21.211697 2308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } }
I20260924 13:38:21.211809 2308 leader_election.cc:302] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f8965302e42491daa2fa5be078bbaac; no voters:
I20260924 13:38:21.212005 2308 leader_election.cc:288] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:21.212077 2311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:21.212265 2308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:21.212445 2311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 1 LEADER]: Becoming Leader. State: Replica: 6f8965302e42491daa2fa5be078bbaac, State: Running, Role: LEADER
I20260924 13:38:21.212549 2311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } }
I20260924 13:38:21.213029 2313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f8965302e42491daa2fa5be078bbaac. Latest consensus state: current_term: 1 leader_uuid: "6f8965302e42491daa2fa5be078bbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } } }
I20260924 13:38:21.213096 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 2240
I20260924 13:38:21.213120 2313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:21.213207 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:21.213284 2312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f8965302e42491daa2fa5be078bbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } } }
I20260924 13:38:21.213346 2312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:21.213462 2319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:21.213508 26561 external_mini_cluster.cc:1428] Setting key 4f91edc964a7205230462e9ca97c044e
I20260924 13:38:21.214026 2319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:21.215845 2319 catalog_manager.cc:1347] Generated new cluster ID: b461c6ee6f2e48268cf125d9ee946afd
I20260924 13:38:21.215903 2319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:21.223670 2319 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:21.224244 2319 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:21.227644 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:21.214171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51940 (local address 127.25.240.126:40193)
0924 13:38:21.214340 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:21.214349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:21.214701 (+ 352us) server_negotiation.cc:407] Connection header received
0924 13:38:21.214820 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:21.214827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:21.214893 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:21.215038 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:21.217681 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.219151 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.220180 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.220829 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.220953 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:21.223771 (+ 2818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:21.223775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:21.223775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:21.223787 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:21.225926 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:21.226317 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:21.226320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:21.226321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:21.226429 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:21.226633 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:21.226634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:21.226634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:21.226921 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:21.227073 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:21.227169 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 13:38:21.227429 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260924 13:38:21.232741 2319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Generated new TSK 0
I20260924 13:38:21.232916 2319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:38:21.265185 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40193
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:21.355489 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:21.355700 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:21.355724 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:21.356920 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:21.356962 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:21.356988 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:21.358500 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:21.358742 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:21.358994 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:21.364331 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:21.364288 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:21.364331 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.364549 2334 server_base.cc:1034] running on GCE node
I20260924 13:38:21.364818 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:21.365061 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:21.366235 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790257101366228 us; error 36 us; skew 500 ppm
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:21.368854 2334 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:38:21.369252 2334 webserver.cc:460] Webserver started at http://127.25.240.65:39191/ using document root <none> and password file <none>
I20260924 13:38:21.369433 2334 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:21.369473 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:21.369637 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:21.370684 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "0e6d4e29075e46b78fe8cbe5ce543d6c"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "285bdf7cf808cb14c834ec207cbafebc"
server_key_iv: "a9642e9d05dcb3c0120cf343be446692"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.370987 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "0e6d4e29075e46b78fe8cbe5ce543d6c"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "285bdf7cf808cb14c834ec207cbafebc"
server_key_iv: "a9642e9d05dcb3c0120cf343be446692"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.372301 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:38:21.373013 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.373178 2334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:21.373262 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "0e6d4e29075e46b78fe8cbe5ce543d6c"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "285bdf7cf808cb14c834ec207cbafebc"
server_key_iv: "a9642e9d05dcb3c0120cf343be446692"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.373318 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:21.402006 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:21.404402 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:21.404518 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:21.404717 2334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:21.405061 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:21.405100 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.405148 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:21.405174 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.410723 2334 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:33187
I20260924 13:38:21.410821 2462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:33187 every 8 connection(s)
I20260924 13:38:21.411120 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:21.420102 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:21.411905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:34723 (local address 127.25.240.126:40193)
0924 13:38:21.411964 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:21.411966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:21.412150 (+ 184us) server_negotiation.cc:407] Connection header received
0924 13:38:21.412645 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:21.412646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:21.412656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:21.412724 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:21.413826 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.414373 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.415395 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.415612 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.415699 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:21.417795 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:21.417796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:21.417796 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:21.417803 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:21.418993 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:21.419340 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:21.419341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:21.419341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:21.419393 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:21.419605 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:21.419606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:21.419606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:21.419700 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:21.419748 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:21.419965 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 13:38:21.420032 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:38:21.420322 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:21.411836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:34723)
0924 13:38:21.412005 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:21.412046 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:21.412562 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:21.412780 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:21.412783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:21.413436 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:21.413706 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:21.413712 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.414470 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.414472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:21.415294 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:21.415296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.415711 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.415712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:21.415842 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:21.416312 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:21.416320 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:21.417715 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:21.419061 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:21.419067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:21.419069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:21.419276 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:21.419447 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:21.419447 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:21.419447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:21.419545 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:21.419750 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:21.419753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:21.419860 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 13:38:21.419939 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:38:21.420069 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260924 13:38:21.420817 2463 heartbeater.cc:347] Connected to a master server at 127.25.240.126:40193
I20260924 13:38:21.420897 2463 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:21.420959 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 2334
I20260924 13:38:21.421065 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:21.421103 2463 heartbeater.cc:511] Master 127.25.240.126:40193 requested a full tablet report, sending...
I20260924 13:38:21.421299 26561 external_mini_cluster.cc:1428] Setting key 0271f556d222e13ee21ec60a5690d496
I20260924 13:38:21.421710 2272 ts_manager.cc:194] Registered new tserver with Master: 0e6d4e29075e46b78fe8cbe5ce543d6c (127.25.240.65:33187)
I20260924 13:38:21.422465 2272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:34723
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:38:21.444164 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40193
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:21.523566 2468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:21.523754 2468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:21.523777 2468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:21.524910 2468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:21.524950 2468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:21.525013 2468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:21.526465 2468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:21.526693 2468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:21.526908 2468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:21.531730 2476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:21.531706 2474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.531749 2468 server_base.cc:1034] running on GCE node
W20260924 13:38:21.531733 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.532198 2468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:21.532387 2468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:21.533520 2468 hybrid_clock.cc:648] HybridClock initialized: now 1790257101533509 us; error 27 us; skew 500 ppm
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:21.535878 2468 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:38:21.536291 2468 webserver.cc:460] Webserver started at http://127.25.240.66:44889/ using document root <none> and password file <none>
I20260924 13:38:21.536507 2468 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:21.536581 2468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:21.536728 2468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:21.537788 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "8504f04a5894498dad6506cf2cb95e1d"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "162eb542c985f9c22ba51bc0b93df204"
server_key_iv: "869adad2d8bb1d6b59259f4188b5a925"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.538137 2468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "8504f04a5894498dad6506cf2cb95e1d"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "162eb542c985f9c22ba51bc0b93df204"
server_key_iv: "869adad2d8bb1d6b59259f4188b5a925"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.539405 2468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:21.540082 2483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.540249 2468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:21.540344 2468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "8504f04a5894498dad6506cf2cb95e1d"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "162eb542c985f9c22ba51bc0b93df204"
server_key_iv: "869adad2d8bb1d6b59259f4188b5a925"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.540423 2468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:21.558627 2468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:21.560434 2468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:21.560600 2468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:21.560822 2468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:21.561122 2468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:21.561177 2468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.561218 2468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:21.561246 2468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.566879 2468 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:43705
I20260924 13:38:21.566922 2596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:43705 every 8 connection(s)
I20260924 13:38:21.567250 2468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:38:21.568573 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 2468
I20260924 13:38:21.568689 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:21.568912 26561 external_mini_cluster.cc:1428] Setting key 3c049f68e3afd3e8018f31ea9317d82e
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:21.577509 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:21.567936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:45295 (local address 127.25.240.126:40193)
0924 13:38:21.568021 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:21.568023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:21.568160 (+ 137us) server_negotiation.cc:407] Connection header received
0924 13:38:21.568759 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:21.568761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:21.568771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:21.568815 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:21.570086 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.570555 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.571739 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.572045 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.572130 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:21.574645 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:21.574647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:21.574647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:21.574653 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:21.576040 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:21.576536 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:21.576537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:21.576537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:21.576597 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:21.576870 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:21.576871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:21.576871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:21.576968 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:21.577015 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:21.577385 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 13:38:21.577448 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 13:38:21.577791 2599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:21.567877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:45295)
0924 13:38:21.568026 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:21.568061 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:21.568657 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:21.568894 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:21.568898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:21.569693 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:21.569996 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:21.570002 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.570650 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.570651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:21.571651 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:21.571653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.572150 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.572151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:21.572281 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:21.572793 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:21.572801 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:21.574535 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:21.576143 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:21.576150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:21.576153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:21.576460 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:21.576664 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:21.576665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:21.576665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:21.576799 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:21.577034 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:21.577040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:21.577232 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 13:38:21.577366 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 13:38:21.577537 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
I20260924 13:38:21.578276 2597 heartbeater.cc:347] Connected to a master server at 127.25.240.126:40193
I20260924 13:38:21.578366 2597 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:21.578564 2597 heartbeater.cc:511] Master 127.25.240.126:40193 requested a full tablet report, sending...
I20260924 13:38:21.578984 2272 ts_manager.cc:194] Registered new tserver with Master: 8504f04a5894498dad6506cf2cb95e1d (127.25.240.66:43705)
I20260924 13:38:21.579346 2272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:45295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:38:21.596118 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40193
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:21.675467 2602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:21.675681 2602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:21.675719 2602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:21.677026 2602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:21.677093 2602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:21.677140 2602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:21.678571 2602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:21.678836 2602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:21.679054 2602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:21.684562 2610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.684545 2602 server_base.cc:1034] running on GCE node
W20260924 13:38:21.684528 2608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:21.684513 2607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.684993 2602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:21.685202 2602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:21.686357 2602 hybrid_clock.cc:648] HybridClock initialized: now 1790257101686322 us; error 48 us; skew 500 ppm
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:21.688668 2602 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:38:21.689164 2602 webserver.cc:460] Webserver started at http://127.25.240.67:39185/ using document root <none> and password file <none>
I20260924 13:38:21.689399 2602 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:21.689464 2602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:21.689621 2602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:21.690701 2602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "0dd15f2ec84f4076b708b9f6efd0c8ac"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "bdf8ba7061d32b5437e766770eb92105"
server_key_iv: "dba68e5b5d1bfa817b006b265b04f7e6"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.691066 2602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "0dd15f2ec84f4076b708b9f6efd0c8ac"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "bdf8ba7061d32b5437e766770eb92105"
server_key_iv: "dba68e5b5d1bfa817b006b265b04f7e6"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.692340 2602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:21.693037 2617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.693203 2602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:21.693298 2602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "0dd15f2ec84f4076b708b9f6efd0c8ac"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "bdf8ba7061d32b5437e766770eb92105"
server_key_iv: "dba68e5b5d1bfa817b006b265b04f7e6"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.693380 2602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:21.729095 2602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:21.731153 2602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:21.731314 2602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:21.731547 2602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:21.731887 2602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:21.731948 2602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.731990 2602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:21.732018 2602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.738066 2602 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:39377
I20260924 13:38:21.738108 2730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:39377 every 8 connection(s)
I20260924 13:38:21.738435 2602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:38:21.741111 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 2602
I20260924 13:38:21.741227 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:21.741407 26561 external_mini_cluster.cc:1428] Setting key 97d2905a4bf9017e1dcd4c5d24930b2f
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:21.747539 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:21.739130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:52573 (local address 127.25.240.126:40193)
0924 13:38:21.739183 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:21.739185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:21.739378 (+ 193us) server_negotiation.cc:407] Connection header received
0924 13:38:21.739883 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:21.739884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:21.739893 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:21.739952 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:21.740913 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.741547 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.742820 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.743045 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.743122 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:21.745171 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:21.745172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:21.745173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:21.745178 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:21.746434 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:21.746796 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:21.746797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:21.746797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:21.746846 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:21.747048 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:21.747049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:21.747049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:21.747149 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:21.747198 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:21.747420 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 13:38:21.747476 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:38:21.747773 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:21.739105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:52573)
0924 13:38:21.739264 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:21.739304 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:21.739791 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:21.740039 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:21.740042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:21.740624 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:21.740833 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:21.740837 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.741658 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.741659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:21.742747 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:21.742749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:21.743135 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:21.743136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:21.743236 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:21.743688 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:21.743696 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:21.745082 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:21.746506 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:21.746511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:21.746512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:21.746731 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:21.746902 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:21.746903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:21.746903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:21.746996 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:21.747202 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:21.747206 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:21.747312 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 13:38:21.747393 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 13:38:21.747505 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260924 13:38:21.748267 2731 heartbeater.cc:347] Connected to a master server at 127.25.240.126:40193
I20260924 13:38:21.748342 2731 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:21.748522 2731 heartbeater.cc:511] Master 127.25.240.126:40193 requested a full tablet report, sending...
I20260924 13:38:21.748907 2272 ts_manager.cc:194] Registered new tserver with Master: 0dd15f2ec84f4076b708b9f6efd0c8ac (127.25.240.67:39377)
I20260924 13:38:21.749293 2272 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:52573
I20260924 13:38:21.753247 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab.
Entry for principal oryx/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab.
WARNING: no policy specified for oryx/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab.
Entry for principal oryx/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab.
WARNING: no policy specified for oryx/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab.
Entry for principal oryx/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab.
I20260924 13:38:21.823983 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 2334
I20260924 13:38:21.829090 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 2468
I20260924 13:38:21.833288 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 2602
I20260924 13:38:21.837323 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 2240
I20260924 13:38:21.842345 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:40193
--webserver_interface=127.25.240.126
--webserver_port=41791
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:40193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:21.929679 2740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:21.929901 2740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:21.929952 2740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:21.931146 2740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:21.931180 2740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:21.931195 2740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:21.931207 2740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:21.931218 2740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:21.932647 2740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:40193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:40193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=41791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:21.932832 2740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:21.933023 2740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:21.938028 2746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:21.938050 2748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.938206 2740 server_base.cc:1034] running on GCE node
W20260924 13:38:21.938050 2745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:21.938547 2740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:21.938879 2740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:21.940040 2740 hybrid_clock.cc:648] HybridClock initialized: now 1790257101940023 us; error 59 us; skew 500 ppm
Sep 24 13:38:21 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257101, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:21.942523 2740 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:21.942981 2740 webserver.cc:460] Webserver started at http://127.25.240.126:41791/ using document root <none> and password file <none>
I20260924 13:38:21.943166 2740 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:21.943217 2740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:21.944464 2740 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:21.945096 2755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:21.945294 2740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:21.945379 2740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "6f8965302e42491daa2fa5be078bbaac"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "65bbc7e34e8d0a781a6c04b683562e64"
server_key_iv: "a6f5e492a1e2bbf4ff6053d071282ae7"
server_key_version: "encryptionkey@0"
I20260924 13:38:21.945771 2740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:21.964826 2740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:21.966770 2740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:21.966882 2740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:21.970837 2740 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:40193
I20260924 13:38:21.970891 2807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:40193 every 8 connection(s)
I20260924 13:38:21.971299 2740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:21.972741 2808 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:38:21.973412 2808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Bootstrap starting.
I20260924 13:38:21.977484 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 2740
I20260924 13:38:21.977792 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:33187
--local_ip_for_outbound_sockets=127.25.240.65
--tserver_master_addrs=127.25.240.126:40193
--webserver_port=39191
--webserver_interface=127.25.240.65
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.240.65
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:38:21.982018 2808 log.cc:824] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:21.983629 2808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:38:21.983908 2808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Bootstrap complete.
I20260924 13:38:21.986183 2808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } }
I20260924 13:38:21.986424 2808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8965302e42491daa2fa5be078bbaac, State: Initialized, Role: FOLLOWER
I20260924 13:38:21.986629 2808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } }
I20260924 13:38:21.986724 2808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:21.986776 2808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:21.986824 2808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:38:21.987776 2808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } }
I20260924 13:38:21.987895 2808 leader_election.cc:302] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f8965302e42491daa2fa5be078bbaac; no voters:
I20260924 13:38:21.988111 2808 leader_election.cc:288] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:38:21.988138 2812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:38:21.988260 2812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [term 2 LEADER]: Becoming Leader. State: Replica: 6f8965302e42491daa2fa5be078bbaac, State: Running, Role: LEADER
I20260924 13:38:21.988376 2808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:21.988418 2812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } }
I20260924 13:38:21.988816 2814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f8965302e42491daa2fa5be078bbaac. Latest consensus state: current_term: 2 leader_uuid: "6f8965302e42491daa2fa5be078bbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } } }
I20260924 13:38:21.988826 2813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6f8965302e42491daa2fa5be078bbaac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8965302e42491daa2fa5be078bbaac" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 40193 } } }
I20260924 13:38:21.989007 2814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:21.989017 2813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:21.989456 2821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:21.990031 2821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:21.990159 2821 catalog_manager.cc:1259] Loaded cluster ID: b461c6ee6f2e48268cf125d9ee946afd
I20260924 13:38:21.990248 2821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:21.991624 2821 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:21.991775 2821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6f8965302e42491daa2fa5be078bbaac: Loaded TSK: 0
I20260924 13:38:21.991974 2821 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:38:22.060613 2810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:22.060822 2810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:22.060846 2810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:22.062119 2810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:22.062162 2810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:22.062191 2810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:22.063602 2810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.65.keytab
--principal=oryx/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:33187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=39191
--tserver_master_addrs=127.25.240.126:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:22.063836 2810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:22.064078 2810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:22.069031 2837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:22.069119 2834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:22.069033 2835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:22.069294 2810 server_base.cc:1034] running on GCE node
I20260924 13:38:22.069510 2810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:22.069921 2810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:22.071134 2810 hybrid_clock.cc:648] HybridClock initialized: now 1790257102071113 us; error 48 us; skew 500 ppm
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257102, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:22.074182 2810 init.cc:377] Logged in from keytab as oryx/127.25.240.65@KRBTEST.COM (short username oryx)
I20260924 13:38:22.074630 2810 webserver.cc:460] Webserver started at http://127.25.240.65:39191/ using document root <none> and password file <none>
I20260924 13:38:22.074864 2810 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:22.074947 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:22.076350 2810 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:22.076982 2844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.077157 2810 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:38:22.077255 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "0e6d4e29075e46b78fe8cbe5ce543d6c"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "285bdf7cf808cb14c834ec207cbafebc"
server_key_iv: "a9642e9d05dcb3c0120cf343be446692"
server_key_version: "encryptionkey@0"
I20260924 13:38:22.077646 2810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:22.095470 2810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:22.097834 2810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:22.097996 2810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:22.098260 2810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:22.098569 2810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:22.098627 2810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.098668 2810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:22.098696 2810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.105120 2810 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:33187
I20260924 13:38:22.105163 2957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:33187 every 8 connection(s)
I20260924 13:38:22.105499 2810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:38:22.113142 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 2810
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.113514 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:43705
--local_ip_for_outbound_sockets=127.25.240.66
--tserver_master_addrs=127.25.240.126:40193
--webserver_port=44889
--webserver_interface=127.25.240.66
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.240.66
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.114862 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.106237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:54627)
0924 13:38:22.106428 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.106485 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.107243 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.107665 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.107670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.108437 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.108737 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.108744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.109858 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.109859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.111095 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.111099 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.111641 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.111642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.111772 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.112257 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.112267 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.114655 (+ 2388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
W20260924 13:38:22.115089 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.106458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:54627 (local address 127.25.240.126:40193)
0924 13:38:22.106618 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.106627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.106665 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:38:22.107434 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.107439 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.107481 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.107589 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.108848 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.109774 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.111180 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.111560 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.111639 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.114782 (+ 3143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:54627: BlockingRecv error: recv got EOF from 127.25.240.65:54627 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260924 13:38:22.115178 2958 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:40193 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.240.126:40193: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.119540 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.115393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:43679)
0924 13:38:22.115448 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.115477 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.115526 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.115678 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.115679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.115826 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.115867 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.115868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.116497 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.116498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.116987 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.116989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.117346 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.117347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.117415 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.117872 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.117876 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.119463 (+ 1587us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 13:38:22.119596 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.115452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:43679 (local address 127.25.240.126:40193)
0924 13:38:22.115525 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.115527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.115536 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:22.115583 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.115584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.115595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.115676 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.115953 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.116417 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.117083 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.117274 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.117348 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.119536 (+ 2188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:43679: BlockingRecv error: recv got EOF from 127.25.240.65:43679 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.124522 2960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.120038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:35631)
0924 13:38:22.120149 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.120161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.120218 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.120340 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.120341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.120506 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.120542 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.120544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.121113 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.121114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.122002 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.122005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.122332 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.122333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.122405 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.122748 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.122753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.124481 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260924 13:38:22.124655 2958 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:38:22.124797 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.120104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:35631 (local address 127.25.240.126:40193)
0924 13:38:22.120154 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.120156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.120203 (+ 47us) server_negotiation.cc:407] Connection header received
0924 13:38:22.120279 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.120280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.120288 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.120330 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.120619 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.121034 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.122084 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.122255 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.122308 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.124588 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:35631: BlockingRecv error: recv got EOF from 127.25.240.65:35631 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 13:38:22.196221 2962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:22.196414 2962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:22.196436 2962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:22.197494 2962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:22.197526 2962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:22.197603 2962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:22.199045 2962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.66.keytab
--principal=oryx/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:43705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=44889
--tserver_master_addrs=127.25.240.126:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:22.199208 2962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:22.199363 2962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:22.204221 2968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:22.204282 2967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:22.204231 2970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:22.204288 2962 server_base.cc:1034] running on GCE node
I20260924 13:38:22.204624 2962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:22.204859 2962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:22.206014 2962 hybrid_clock.cc:648] HybridClock initialized: now 1790257102206005 us; error 46 us; skew 500 ppm
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257102, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:22.208390 2962 init.cc:377] Logged in from keytab as oryx/127.25.240.66@KRBTEST.COM (short username oryx)
I20260924 13:38:22.208760 2962 webserver.cc:460] Webserver started at http://127.25.240.66:44889/ using document root <none> and password file <none>
I20260924 13:38:22.208927 2962 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:22.208977 2962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:22.210335 2962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:22.210913 2977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.211105 2962 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:22.211185 2962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "8504f04a5894498dad6506cf2cb95e1d"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "162eb542c985f9c22ba51bc0b93df204"
server_key_iv: "869adad2d8bb1d6b59259f4188b5a925"
server_key_version: "encryptionkey@0"
I20260924 13:38:22.211472 2962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:22.225323 2962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:22.227108 2962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:22.227213 2962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:22.227386 2962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:22.227629 2962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:22.227659 2962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.227680 2962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:22.227692 2962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.233260 2962 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:43705
I20260924 13:38:22.233353 3090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:43705 every 8 connection(s)
I20260924 13:38:22.233733 2962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:38:22.239569 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 2962
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.239882 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:39377
--local_ip_for_outbound_sockets=127.25.240.67
--tserver_master_addrs=127.25.240.126:40193
--webserver_port=39185
--webserver_interface=127.25.240.67
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.240.67
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:38:22.241381 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.234458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:53245 (local address 127.25.240.126:40193)
0924 13:38:22.234547 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.234550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.234758 (+ 208us) server_negotiation.cc:407] Connection header received
0924 13:38:22.235280 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.235282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.235299 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.235351 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.236276 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.236777 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.237838 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.238050 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.238143 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.241298 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:53245: BlockingRecv error: recv got EOF from 127.25.240.66:53245 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 13:38:22.241413 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.234416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:53245)
0924 13:38:22.234595 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.234667 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.235168 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.235416 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.235420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.235984 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.236177 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.236181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.236873 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.236874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.237751 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.237753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.238152 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.238153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.238285 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.238800 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.238808 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.241186 (+ 2378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
W20260924 13:38:22.241694 3091 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:40193 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.240.126:40193: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.246783 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.241945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:37259)
0924 13:38:22.242007 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.242021 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.242082 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.242207 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.242208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.242366 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.242422 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.242424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.243306 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.243307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.243870 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.243871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.244381 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.244381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.244429 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.244875 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.244879 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.246684 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 13:38:22.246804 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.241980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:37259 (local address 127.25.240.126:40193)
0924 13:38:22.242018 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.242020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.242086 (+ 66us) server_negotiation.cc:407] Connection header received
0924 13:38:22.242133 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.242133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.242141 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.242185 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.242502 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.243229 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.243957 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.244301 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.244381 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.246760 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:37259: BlockingRecv error: recv got EOF from 127.25.240.66:37259 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.251772 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.247058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:43073)
0924 13:38:22.247177 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.247189 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.247245 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.247362 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.247362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.247515 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.247555 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.247556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.248142 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.248143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.249092 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.249095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.249417 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.249418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.249483 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.249856 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.249862 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.251695 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260924 13:38:22.251824 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.247133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:43073 (local address 127.25.240.126:40193)
0924 13:38:22.247189 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.247191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.247229 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:38:22.247304 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.247305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.247312 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.247357 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.247633 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.248063 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.249177 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.249364 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.249414 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.251760 (+ 2346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:43073: BlockingRecv error: recv got EOF from 127.25.240.66:43073 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 13:38:22.251893 3091 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:38:22.324592 3094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:22.324822 3094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:22.324862 3094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:22.326077 3094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:22.326140 3094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:22.326185 3094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:22.327603 3094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/krb5kdc/oryx_127.25.240.67.keytab
--principal=oryx/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:39377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=39185
--tserver_master_addrs=127.25.240.126:40193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:22.327867 3094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:22.328083 3094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:22.332924 3100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:22.332960 3099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:22.332926 3102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:22.333233 3094 server_base.cc:1034] running on GCE node
I20260924 13:38:22.333455 3094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:22.333714 3094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:22.334877 3094 hybrid_clock.cc:648] HybridClock initialized: now 1790257102334854 us; error 36 us; skew 500 ppm
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257102, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:22.337361 3094 init.cc:377] Logged in from keytab as oryx/127.25.240.67@KRBTEST.COM (short username oryx)
I20260924 13:38:22.337841 3094 webserver.cc:460] Webserver started at http://127.25.240.67:39185/ using document root <none> and password file <none>
I20260924 13:38:22.338064 3094 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:22.338150 3094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:22.339476 3094 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:22.340044 3109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.340232 3094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:22.340341 3094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "0dd15f2ec84f4076b708b9f6efd0c8ac"
format_stamp: "Formatted at 2026-09-24 13:38:21 on dist-test-slave-m5z1"
server_key: "bdf8ba7061d32b5437e766770eb92105"
server_key_iv: "dba68e5b5d1bfa817b006b265b04f7e6"
server_key_version: "encryptionkey@0"
I20260924 13:38:22.340692 3094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:22.367746 3094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:22.370306 3094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:22.370482 3094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:22.370738 3094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:22.371052 3094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:22.371112 3094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.371156 3094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:22.371181 3094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:22.376869 3094 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:39377
I20260924 13:38:22.376919 3222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:39377 every 8 connection(s)
I20260924 13:38:22.377300 3094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 13:38:22.384590 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.378119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:41775 (local address 127.25.240.126:40193)
0924 13:38:22.378202 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.378204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.378382 (+ 178us) server_negotiation.cc:407] Connection header received
0924 13:38:22.378907 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.378908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.378920 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.378965 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.379896 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.380452 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.381483 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.381737 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.381822 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.384533 (+ 2711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:41775: BlockingRecv error: recv got EOF from 127.25.240.67:41775 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:38:22.384591 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.378086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:41775)
0924 13:38:22.378261 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.378296 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.378799 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.379036 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.379040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.379603 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.379790 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.379795 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.380542 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.380543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.381389 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.381392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.381830 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.381831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.381959 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.382412 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.382420 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.384446 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
W20260924 13:38:22.384843 3223 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:40193 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.240.126:40193: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
I20260924 13:38:22.385790 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 3094
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.389763 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.385073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:35165)
0924 13:38:22.385127 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.385138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.385177 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.385389 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.385389 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.385495 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.385537 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.385538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.386436 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.386437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.386916 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.386917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.387337 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.387338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.387403 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.387769 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.387774 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.389717 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 13:38:22.389889 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.385237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:35165 (local address 127.25.240.126:40193)
0924 13:38:22.385290 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.385292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.385299 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:38:22.385317 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.385317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.385324 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.385371 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.385678 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.386364 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.386990 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.387271 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.387318 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.389815 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:35165: BlockingRecv error: recv got EOF from 127.25.240.67:35165 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.393864 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.386186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.25.240.126:40193)
0924 13:38:22.386471 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.386473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.386478 (+ 5us) server_negotiation.cc:407] Connection header received
0924 13:38:22.386501 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.386502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.386510 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.386578 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.387184 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.387845 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.388727 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.388945 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.389020 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.390445 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:22.390451 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:22.390451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:22.390466 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:22.392338 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:22.392724 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:22.392728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:22.392728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:22.392831 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:22.393061 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:22.393062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:22.393062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:22.393333 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:22.393482 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:22.393615 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 13:38:22.393735 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":56,"threads_started":1}
Sep 24 13:38:22 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:22.394691 3225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:22.390178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:35947)
0924 13:38:22.390228 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:22.390238 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:22.390272 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:22.390419 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:22.390420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:22.390563 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:22.390593 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.390595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.391369 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.391370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.392124 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:22.392127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.392498 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.392499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:22.392536 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.392904 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:22.392909 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:22.394614 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 13:38:22.394740 2961 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:22.390244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:35947 (local address 127.25.240.126:40193)
0924 13:38:22.390277 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:22.390279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:22.390291 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:38:22.390317 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:22.390318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:22.390327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:22.390391 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:22.390669 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.391282 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.392211 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:22.392420 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:22.392492 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:22.394685 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:35947: BlockingRecv error: recv got EOF from 127.25.240.67:35947 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:38:22.394807 3223 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 13:38:23 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:23 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:23.133560 3228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:23.125072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:37425)
0924 13:38:23.125233 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:23.125245 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:23.125300 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:23.125446 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:23.125447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:23.125626 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:23.125675 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:23.125677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.126393 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.126394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:23.126933 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:23.126935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.127263 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.127264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:23.127306 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:23.131660 (+ 4354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:23.131665 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:23.133454 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
W20260924 13:38:23.133656 3229 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:23.125178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:37425 (local address 127.25.240.126:40193)
0924 13:38:23.125301 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:23.125303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:23.125317 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:38:23.125360 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:23.125360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:23.125370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:23.125427 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:23.125780 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.126305 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.127012 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.127198 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.127246 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:23.133536 (+ 6290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:37425: BlockingRecv error: recv got EOF from 127.25.240.65:37425 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
Sep 24 13:38:23 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:23 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:23.256927 3230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:23.252245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:51795)
0924 13:38:23.252414 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:23.252426 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:23.252501 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:23.252676 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:23.252677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:23.252860 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:23.252919 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:23.252921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.253626 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.253627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:23.254186 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:23.254188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.254567 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.254568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:23.254625 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:23.255077 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:23.255082 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:23.256820 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
W20260924 13:38:23.256965 3229 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:23.252329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:51795 (local address 127.25.240.126:40193)
0924 13:38:23.252402 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:23.252405 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:23.252497 (+ 92us) server_negotiation.cc:407] Connection header received
0924 13:38:23.252558 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:23.252559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:23.252572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:23.252665 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:23.253013 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.253513 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.254286 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.254484 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.254558 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:23.256900 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:51795: BlockingRecv error: recv got EOF from 127.25.240.66:51795 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 13:38:23 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:23 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:23.399829 3231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:23.395176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:36051)
0924 13:38:23.395310 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:23.395329 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:23.395388 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:23.395520 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:23.395521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:23.395682 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:23.395730 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:23.395732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.396350 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.396351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:23.396881 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:23.396882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.397203 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.397204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:23.397242 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:23.397694 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:23.397698 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:23.399724 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
W20260924 13:38:23.399858 3229 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:23.395246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:36051 (local address 127.25.240.126:40193)
0924 13:38:23.395314 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:23.395316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:23.395378 (+ 62us) server_negotiation.cc:407] Connection header received
0924 13:38:23.395435 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:23.395435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:23.395447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:23.395501 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:23.395797 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.396284 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.396954 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:23.397141 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:23.397191 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:23.399794 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:36051: BlockingRecv error: recv got EOF from 127.25.240.67:36051 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 13:38:24 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:24 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:24.138445 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:24.134045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:57549)
0924 13:38:24.134178 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:24.134189 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:24.134251 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:24.134380 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:24.134381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:24.134556 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:24.134608 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:24.134610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.135258 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.135259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:24.135780 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:24.135781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.136194 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.136194 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:24.136237 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:24.136665 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:24.136670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:24.138387 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":77,"threads_started":1}
W20260924 13:38:24.138576 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:24.134090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:57549 (local address 127.25.240.126:40193)
0924 13:38:24.134214 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:24.134216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:24.134242 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:38:24.134296 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:24.134297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:24.134306 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:24.134369 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:24.134682 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.135188 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.135863 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.136120 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.136186 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:24.138487 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:57549: BlockingRecv error: recv got EOF from 127.25.240.65:57549 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
Sep 24 13:38:24 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:24 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:24.261951 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:24.257387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:40469)
0924 13:38:24.257508 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:24.257518 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:24.257600 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:24.257771 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:24.257772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:24.257928 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:24.257978 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:24.257980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.258590 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.258591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:24.259104 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:24.259106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.259411 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.259412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:24.259449 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:24.259931 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:24.259943 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:24.261860 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":51,"threads_started":1}
W20260924 13:38:24.262013 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:24.257513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:40469 (local address 127.25.240.126:40193)
0924 13:38:24.257612 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:24.257614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:24.257659 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:38:24.257683 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:24.257684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:24.257695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:24.257754 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:24.258042 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.258528 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.259171 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.259350 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.259397 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:24.261966 (+ 2569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:40469: BlockingRecv error: recv got EOF from 127.25.240.66:40469 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 13:38:24 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:24 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:24.404866 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:24.400393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:43083)
0924 13:38:24.400531 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:24.400542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:24.400606 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:24.400786 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:24.400787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:24.400959 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:24.401006 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:24.401007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.401836 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.401837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:24.402364 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:24.402365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.402715 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.402716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:24.402754 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:24.403187 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:24.403191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:24.404809 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
W20260924 13:38:24.404985 3233 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:24.400532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:43083 (local address 127.25.240.126:40193)
0924 13:38:24.400608 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:24.400611 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:24.400629 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:38:24.400684 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:24.400685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:24.400699 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:24.400749 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:24.401106 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.401766 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.402438 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:24.402615 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:24.402712 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:24.404906 (+ 2194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:43083: BlockingRecv error: recv got EOF from 127.25.240.67:43083 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 13:38:25 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:25 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:25.143549 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:25.138915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:42281)
0924 13:38:25.139054 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:25.139064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:25.139113 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:25.139255 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:25.139256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:25.139410 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:25.139460 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:25.139461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.140090 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.140091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:25.140601 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:25.140603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.140922 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.140923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:25.140965 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:25.141496 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:25.141503 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:25.143421 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260924 13:38:25.143560 3237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:25.138991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:42281 (local address 127.25.240.126:40193)
0924 13:38:25.139108 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:25.139111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:25.139150 (+ 39us) server_negotiation.cc:407] Connection header received
0924 13:38:25.139172 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:25.139172 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:25.139185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:25.139237 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:25.139523 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.140025 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.140676 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.140858 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.140906 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:25.143508 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:42281: BlockingRecv error: recv got EOF from 127.25.240.65:42281 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
Sep 24 13:38:25 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:25 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:25.266891 3238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:25.262503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:42869)
0924 13:38:25.262653 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:25.262664 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:25.262732 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:25.262871 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:25.262872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:25.263035 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:25.263086 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:25.263088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.263711 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.263712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:25.264224 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:25.264225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.264536 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.264536 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:25.264573 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:25.265086 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:25.265092 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:25.266824 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
W20260924 13:38:25.267076 3237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:25.262578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:42869 (local address 127.25.240.126:40193)
0924 13:38:25.262677 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:25.262679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:25.262735 (+ 56us) server_negotiation.cc:407] Connection header received
0924 13:38:25.262792 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:25.262793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:25.262807 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:25.262849 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:25.263150 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.263636 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.264294 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.264474 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.264522 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:25.266934 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:42869: BlockingRecv error: recv got EOF from 127.25.240.66:42869 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 13:38:25 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:25 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:25.409885 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:25.405440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:53927)
0924 13:38:25.405641 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:25.405669 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:25.405749 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:25.405894 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:25.405894 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:25.406048 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:25.406098 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:25.406099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.406719 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.406720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:25.407224 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:25.407225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.407528 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.407529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:25.407565 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:25.407981 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:25.407985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:25.409780 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":108,"threads_started":1}
W20260924 13:38:25.410069 3237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:25.405527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:53927 (local address 127.25.240.126:40193)
0924 13:38:25.405601 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:25.405603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:25.405757 (+ 154us) server_negotiation.cc:407] Connection header received
0924 13:38:25.405800 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:25.405801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:25.405817 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:25.405879 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:25.406159 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.406650 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.407291 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:25.407466 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:25.407516 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:25.409906 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:53927: BlockingRecv error: recv got EOF from 127.25.240.67:53927 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 13:38:26 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:26 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:26.148619 3240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:26.144073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:38845)
0924 13:38:26.144204 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:26.144214 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:26.144270 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:26.144395 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:26.144396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:26.144562 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:26.144614 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:26.144615 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.145272 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.145273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:26.145881 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:26.145884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.146234 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.146234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:26.146283 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:26.146701 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:26.146706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:26.148520 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260924 13:38:26.148669 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:26.144143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:38845 (local address 127.25.240.126:40193)
0924 13:38:26.144284 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:26.144286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:26.144296 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:26.144317 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:26.144318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:26.144327 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:26.144390 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:26.144687 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.145192 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.145959 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.146162 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.146221 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:26.148598 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:38845: BlockingRecv error: recv got EOF from 127.25.240.65:38845 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
Sep 24 13:38:26 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:26 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:26.272215 3242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:26.267420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:41211)
0924 13:38:26.267610 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:26.267622 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:26.267713 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:26.267870 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:26.267871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:26.268051 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:26.268099 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:26.268101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.268784 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.268785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:26.269318 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:26.269320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.269760 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.269761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:26.269820 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:26.270244 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:26.270248 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:26.272108 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
W20260924 13:38:26.272256 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:26.267511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:41211 (local address 127.25.240.126:40193)
0924 13:38:26.267592 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:26.267594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:26.267701 (+ 107us) server_negotiation.cc:407] Connection header received
0924 13:38:26.267773 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:26.267774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:26.267788 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:26.267865 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:26.268187 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.268698 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.269407 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.269685 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.269762 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:26.272195 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:41211: BlockingRecv error: recv got EOF from 127.25.240.66:41211 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 13:38:26 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:26 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:26.415040 3243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:26.410331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:60951)
0924 13:38:26.410479 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:26.410490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:26.410549 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:26.410679 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:26.410680 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:26.410872 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:26.410931 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:26.410933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.411779 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.411779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:26.412321 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:26.412323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.412628 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.412628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:26.412665 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:26.413167 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:26.413173 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:26.414978 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260924 13:38:26.415134 3241 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:26.410414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:60951 (local address 127.25.240.126:40193)
0924 13:38:26.410482 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:26.410489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:26.410549 (+ 60us) server_negotiation.cc:407] Connection header received
0924 13:38:26.410592 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:26.410593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:26.410604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:26.410659 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:26.411012 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.411711 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.412397 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:26.412569 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:26.412615 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:26.415066 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:60951: BlockingRecv error: recv got EOF from 127.25.240.67:60951 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 13:38:27 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:27 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:27.154098 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:27.149102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:35265)
0924 13:38:27.149369 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:27.149386 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:27.149453 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:27.149659 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:27.149660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:27.149839 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:27.149889 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:27.149890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.150521 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.150522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:27.151052 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:27.151053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.151469 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.151470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:27.151526 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:27.151994 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:27.151998 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:27.154009 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":66,"threads_started":1}
W20260924 13:38:27.154162 3245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:27.149135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:35265 (local address 127.25.240.126:40193)
0924 13:38:27.149392 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:27.149394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:27.149432 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:38:27.149508 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:27.149508 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:27.149519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:27.149638 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:27.149953 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.150452 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.151126 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.151318 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.151370 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:27.154098 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:35265: BlockingRecv error: recv got EOF from 127.25.240.65:35265 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":193,"threads_started":1}
Sep 24 13:38:27 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:27 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:27.277333 3246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:27.272716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:55507)
0924 13:38:27.272933 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:27.272944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:27.273025 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:27.273189 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:27.273190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:27.273410 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:27.273460 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:27.273461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.274139 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.274140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:27.274648 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:27.274649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.274976 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.274977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:27.275013 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:27.275519 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:27.275523 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:27.277275 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":70,"threads_started":1}
W20260924 13:38:27.277443 3245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:27.272843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:55507 (local address 127.25.240.126:40193)
0924 13:38:27.272885 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:27.272887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:27.273011 (+ 124us) server_negotiation.cc:407] Connection header received
0924 13:38:27.273096 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:27.273097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:27.273110 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:27.273168 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:27.273529 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.274070 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.274724 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.274910 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.274963 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:27.277369 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:55507: BlockingRecv error: recv got EOF from 127.25.240.66:55507 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 13:38:27 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:27 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:27.420058 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:27.415504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:49787)
0924 13:38:27.415647 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:27.415657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:27.415745 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:27.415899 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:27.415900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:27.416079 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:27.416123 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:27.416125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.416797 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.416798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:27.417325 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:27.417327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.417753 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.417754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:27.417813 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:27.418237 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:27.418241 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:27.420000 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
W20260924 13:38:27.420176 3245 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:27.415563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:49787 (local address 127.25.240.126:40193)
0924 13:38:27.415611 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:27.415613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:27.415728 (+ 115us) server_negotiation.cc:407] Connection header received
0924 13:38:27.415802 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:27.415803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:27.415818 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:27.415893 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:27.416215 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.416710 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.417411 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:27.417679 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:27.417754 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:27.420098 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:49787: BlockingRecv error: recv got EOF from 127.25.240.67:49787 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 13:38:28 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:28 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:28.163697 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:28.154612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:43633)
0924 13:38:28.154760 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:28.154770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:28.154827 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:28.154958 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:28.154959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:28.155122 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:28.155186 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:28.155187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.155816 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.155817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:28.156341 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:28.156342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.156657 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.156658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:28.156707 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:28.161832 (+ 5125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:28.161837 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:28.163637 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
W20260924 13:38:28.163830 3249 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:28.154702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:43633 (local address 127.25.240.126:40193)
0924 13:38:28.154828 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:28.154830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:28.154857 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:38:28.154879 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:28.154879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:28.154889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:28.154938 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:28.155256 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.155754 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.156407 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.156594 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.156643 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:28.163742 (+ 7099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:43633: BlockingRecv error: recv got EOF from 127.25.240.65:43633 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
Sep 24 13:38:28 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:28 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:28.282604 3250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:28.277829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:46267)
0924 13:38:28.277948 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:28.277960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:28.278049 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:28.278197 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:28.278198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:28.278368 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:28.278425 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:28.278427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.279102 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.279103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:28.279625 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:28.279627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.279996 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.279997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:28.280072 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:28.280635 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:28.280640 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:28.282511 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
W20260924 13:38:28.282657 3249 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:28.277892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:46267 (local address 127.25.240.126:40193)
0924 13:38:28.277942 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:28.277945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:28.278034 (+ 89us) server_negotiation.cc:407] Connection header received
0924 13:38:28.278121 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:28.278122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:28.278134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:28.278182 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:28.278505 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.279008 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.279719 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.279920 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.279998 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:28.282592 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:46267: BlockingRecv error: recv got EOF from 127.25.240.66:46267 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 13:38:28 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:28 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:28.425086 3251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:28.420654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:48667)
0924 13:38:28.420805 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:28.420821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:28.420877 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:28.421007 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:28.421008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:28.421168 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:28.421229 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:28.421231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.421918 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.421919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:28.422426 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:28.422427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.422751 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.422752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:28.422802 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:28.423301 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:28.423305 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:28.425026 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
W20260924 13:38:28.425222 3249 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:28.420723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:48667 (local address 127.25.240.126:40193)
0924 13:38:28.420805 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:28.420807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:28.420881 (+ 74us) server_negotiation.cc:407] Connection header received
0924 13:38:28.420920 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:28.420921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:28.420933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:28.420989 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:28.421294 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.421850 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.422496 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:28.422677 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:28.422734 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:28.425138 (+ 2404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:48667: BlockingRecv error: recv got EOF from 127.25.240.67:48667 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 13:38:29 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:29 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:29.168763 3252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:29.164274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:51227)
0924 13:38:29.164446 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:29.164456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:29.164506 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:29.164659 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:29.164659 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:29.164833 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:29.164881 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:29.164883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.165505 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.165506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:29.166089 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:29.166092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.166411 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.166412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:29.166449 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:29.166934 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:29.166939 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:29.168699 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":112,"threads_started":1}
W20260924 13:38:29.168906 3253 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:29.164331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:51227 (local address 127.25.240.126:40193)
0924 13:38:29.164479 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:29.164481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:29.164493 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:38:29.164563 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:29.164564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:29.164574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:29.164635 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:29.164951 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.165439 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.166165 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.166350 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.166398 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:29.168816 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:51227: BlockingRecv error: recv got EOF from 127.25.240.65:51227 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
Sep 24 13:38:29 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:29 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:29.287971 3254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:29.283320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:42733)
0924 13:38:29.283490 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:29.283502 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:29.283574 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:29.283746 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:29.283747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:29.283927 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:29.283973 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:29.283975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.284656 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.284657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:29.285192 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:29.285194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.285597 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.285598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:29.285651 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:29.286110 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:29.286114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:29.287871 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":82,"threads_started":1}
W20260924 13:38:29.288024 3253 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:29.283404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:42733 (local address 127.25.240.126:40193)
0924 13:38:29.283491 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:29.283493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:29.283554 (+ 61us) server_negotiation.cc:407] Connection header received
0924 13:38:29.283633 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:29.283634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:29.283646 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:29.283720 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:29.284058 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.284567 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.285286 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.285488 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.285615 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:29.287958 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:42733: BlockingRecv error: recv got EOF from 127.25.240.66:42733 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 13:38:29 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:29 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:29.429965 3255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:29.425676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:46005)
0924 13:38:29.425813 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:29.425823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:29.425878 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:29.426006 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:29.426006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:29.426167 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:29.426215 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:29.426217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.426832 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.426833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:29.427345 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:29.427346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.427648 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.427648 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:29.427687 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:29.428109 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:29.428114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:29.429905 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
W20260924 13:38:29.430068 3253 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:29.425760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:46005 (local address 127.25.240.126:40193)
0924 13:38:29.425818 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:29.425820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:29.425865 (+ 45us) server_negotiation.cc:407] Connection header received
0924 13:38:29.425934 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:29.425935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:29.425946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:29.425988 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:29.426275 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.426768 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.427412 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:29.427587 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:29.427634 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:29.429998 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:46005: BlockingRecv error: recv got EOF from 127.25.240.67:46005 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 13:38:30 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:30 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:30.173784 3256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:30.169225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:38259)
0924 13:38:30.169470 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:30.169486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:30.169589 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:30.169769 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:30.169770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:30.169934 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:30.169984 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:30.169986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.170609 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.170610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:30.171127 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:30.171128 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.171451 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.171452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:30.171490 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:30.171928 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:30.171932 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:30.173722 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":86,"threads_started":1}
W20260924 13:38:30.173911 3257 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:30.169447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:38259 (local address 127.25.240.126:40193)
0924 13:38:30.169598 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:30.169600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:30.169608 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:30.169659 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:30.169660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:30.169674 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:30.169732 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:30.170048 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.170544 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.171195 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.171389 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.171438 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:30.173825 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:38259: BlockingRecv error: recv got EOF from 127.25.240.65:38259 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
Sep 24 13:38:30 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:30 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:30.293480 3258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:30.288473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:41889)
0924 13:38:30.288608 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:30.288618 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:30.288672 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:30.288820 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:30.288820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:30.288979 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:30.289028 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:30.289030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.289991 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.289992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:30.290508 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:30.290509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.290825 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.290826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:30.290865 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:30.291305 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:30.291310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:30.293422 (+ 2112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
W20260924 13:38:30.293613 3257 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:30.288543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:41889 (local address 127.25.240.126:40193)
0924 13:38:30.288606 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:30.288608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:30.288709 (+ 101us) server_negotiation.cc:407] Connection header received
0924 13:38:30.288735 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:30.288736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:30.288748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:30.288789 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:30.289089 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.289923 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.290581 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.290764 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.290811 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:30.293514 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:41889: BlockingRecv error: recv got EOF from 127.25.240.66:41889 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 13:38:30 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:30 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:30.435123 3259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:30.430640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:35749)
0924 13:38:30.430797 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:30.430808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:30.430856 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:30.430999 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:30.431000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:30.431153 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:30.431201 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:30.431202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.431826 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.431827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:30.432332 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:30.432333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.432661 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.432663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:30.432713 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:30.433148 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:30.433153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:30.434986 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260924 13:38:30.435125 3257 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:30.430813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:35749 (local address 127.25.240.126:40193)
0924 13:38:30.430880 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:30.430882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:30.430892 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:30.430922 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:30.430923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:30.430935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:30.430993 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:30.431268 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.431757 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.432399 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:30.432574 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:30.432624 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:30.435078 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:35749: BlockingRecv error: recv got EOF from 127.25.240.67:35749 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 13:38:31 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:31 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:31.178913 3260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:31.174298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:48505)
0924 13:38:31.174455 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:31.174466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:31.174523 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:31.174645 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:31.174646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:31.174812 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:31.174861 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:31.174863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.175495 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.175496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:31.176005 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:31.176007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.176329 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.176330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:31.176369 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:31.176863 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:31.176871 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:31.178853 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
W20260924 13:38:31.179018 3261 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:31.174384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:48505 (local address 127.25.240.126:40193)
0924 13:38:31.174523 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:31.174525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:31.174534 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:31.174560 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:31.174561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:31.174571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:31.174630 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:31.174931 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.175430 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.176077 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.176269 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.176319 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:31.178939 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:48505: BlockingRecv error: recv got EOF from 127.25.240.65:48505 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":55,"threads_started":1}
Sep 24 13:38:31 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:31 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:31.298496 3262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:31.293972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:35875)
0924 13:38:31.294121 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:31.294130 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:31.294187 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:31.294326 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:31.294327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:31.294490 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:31.294537 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:31.294538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.295159 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.295160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:31.295671 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:31.295673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.296026 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.296027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:31.296091 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:31.296613 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:31.296618 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:31.298437 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
W20260924 13:38:31.298600 3261 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:31.294040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:35875 (local address 127.25.240.126:40193)
0924 13:38:31.294104 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:31.294106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:31.294187 (+ 81us) server_negotiation.cc:407] Connection header received
0924 13:38:31.294246 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:31.294247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:31.294258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:31.294307 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:31.294596 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.295091 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.295741 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.295928 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.296009 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:31.298528 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:35875: BlockingRecv error: recv got EOF from 127.25.240.66:35875 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 13:38:31 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:31 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:31.440434 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:31.435585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:54501)
0924 13:38:31.435721 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:31.435731 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:31.435824 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:31.435966 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:31.435967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:31.436125 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:31.436173 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:31.436174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.437128 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.437129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:31.437677 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:31.437679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.438000 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.438001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:31.438045 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:31.438462 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:31.438466 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:31.440335 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
W20260924 13:38:31.440479 3261 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:31.435660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:54501 (local address 127.25.240.126:40193)
0924 13:38:31.435725 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:31.435727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:31.435820 (+ 93us) server_negotiation.cc:407] Connection header received
0924 13:38:31.435870 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:31.435871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:31.435883 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:31.435948 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:31.436406 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.437054 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.437747 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:31.437932 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:31.437985 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:31.440415 (+ 2430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:54501: BlockingRecv error: recv got EOF from 127.25.240.67:54501 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 13:38:32 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:32 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:32.184258 3264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:32.179587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:34765)
0924 13:38:32.179786 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:32.179799 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:32.179864 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:32.180067 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:32.180068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:32.180254 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:32.180335 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:32.180337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.181038 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.181040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:32.181600 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:32.181602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.181959 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.181960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:32.182019 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:32.182491 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:32.182497 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:32.184202 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
W20260924 13:38:32.184399 3265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:32.179678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:34765 (local address 127.25.240.126:40193)
0924 13:38:32.179852 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:32.179855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:32.179867 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:38:32.179969 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:32.179969 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:32.179979 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:32.180060 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:32.180402 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.180964 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.181683 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.181887 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.181962 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:32.184306 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:34765: BlockingRecv error: recv got EOF from 127.25.240.65:34765 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
Sep 24 13:38:32 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:32 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:32.303156 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:32.298973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:52653)
0924 13:38:32.299096 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:32.299111 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:32.299166 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:32.299298 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:32.299299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:32.299460 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:32.299510 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:32.299511 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.300123 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.300124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:32.300635 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:32.300637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.300951 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.300952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:32.300993 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:32.301426 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:32.301430 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:32.303099 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
W20260924 13:38:32.303252 3265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:32.299037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:52653 (local address 127.25.240.126:40193)
0924 13:38:32.299090 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:32.299092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:32.299154 (+ 62us) server_negotiation.cc:407] Connection header received
0924 13:38:32.299210 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:32.299210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:32.299220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:32.299284 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:32.299571 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.300058 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.300703 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.300888 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.300937 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:32.303189 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:52653: BlockingRecv error: recv got EOF from 127.25.240.66:52653 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 13:38:32 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:32 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:32.445418 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:32.441084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:36295)
0924 13:38:32.441253 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:32.441267 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:32.441338 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:32.441466 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:32.441467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:32.441652 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:32.441701 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:32.441703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.442316 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.442317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:32.442827 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:32.442828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.443156 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.443156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:32.443201 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:32.443609 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:32.443613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:32.445359 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
W20260924 13:38:32.445510 3265 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:32.441231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:36295 (local address 127.25.240.126:40193)
0924 13:38:32.441299 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:32.441301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:32.441322 (+ 21us) server_negotiation.cc:407] Connection header received
0924 13:38:32.441395 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:32.441395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:32.441407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:32.441448 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:32.441759 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.442250 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.442906 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:32.443097 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:32.443144 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:32.445446 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:36295: BlockingRecv error: recv got EOF from 127.25.240.67:36295 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 13:38:33 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:33 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:33.194952 3268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:33.184736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:57809)
0924 13:38:33.184874 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:33.184886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:33.184966 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:33.185143 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:33.185144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:33.185321 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:33.185368 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:33.185370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.186114 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.186116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:33.186644 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:33.186646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.187020 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.187021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:33.187088 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:33.193059 (+ 5971us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:33.193073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:33.194880 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260924 13:38:33.195103 3269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:33.184813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:57809 (local address 127.25.240.126:40193)
0924 13:38:33.184975 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:33.184977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:33.184988 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:38:33.185026 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:33.185027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:33.185038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:33.185107 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:33.185450 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.186028 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.186726 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.186941 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.187009 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:33.195007 (+ 7998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:57809: BlockingRecv error: recv got EOF from 127.25.240.65:57809 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":93,"threads_started":1}
Sep 24 13:38:33 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:33 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:33.308719 3270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:33.303871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:51739)
0924 13:38:33.304047 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:33.304058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:33.304121 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:33.304249 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:33.304249 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:33.304411 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:33.304476 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:33.304478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.305084 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.305085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:33.305646 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:33.305648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.305956 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.305957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:33.306000 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:33.306558 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:33.306564 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:33.308606 (+ 2042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
W20260924 13:38:33.308758 3269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:33.303928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:51739 (local address 127.25.240.126:40193)
0924 13:38:33.303994 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:33.303997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:33.304123 (+ 126us) server_negotiation.cc:407] Connection header received
0924 13:38:33.304169 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:33.304169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:33.304181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:33.304231 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:33.304535 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.305022 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.305712 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.305899 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.305947 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:33.308689 (+ 2742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:51739: BlockingRecv error: recv got EOF from 127.25.240.66:51739 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 13:38:33 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:33 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:33.450660 3271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:33.445881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:47325)
0924 13:38:33.446052 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:33.446063 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:33.446146 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:33.446295 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:33.446295 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:33.446485 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:33.446531 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:33.446533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.447179 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.447180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:33.447754 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:33.447756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.448137 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.448138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:33.448190 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:33.448624 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:33.448628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:33.450591 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":107,"threads_started":1}
W20260924 13:38:33.450766 3269 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:33.445960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:47325 (local address 127.25.240.126:40193)
0924 13:38:33.446033 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:33.446035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:33.446127 (+ 92us) server_negotiation.cc:407] Connection header received
0924 13:38:33.446220 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:33.446221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:33.446232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:33.446289 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:33.446607 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.447098 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.447858 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:33.448047 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:33.448139 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:33.450677 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:47325: BlockingRecv error: recv got EOF from 127.25.240.67:47325 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 13:38:34 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:34 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:34.199702 3272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:34.195402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:55895)
0924 13:38:34.195548 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:34.195561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:34.195620 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:34.195751 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:34.195752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:34.195918 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:34.195967 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:34.195968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.196606 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.196607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:34.197118 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:34.197119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.197445 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.197446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:34.197484 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:34.197964 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:34.197969 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:34.199642 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
W20260924 13:38:34.199820 3273 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:34.195459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:55895 (local address 127.25.240.126:40193)
0924 13:38:34.195599 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:34.195601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:34.195609 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:34.195676 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:34.195677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:34.195687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:34.195736 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:34.196032 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.196541 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.197184 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.197386 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.197437 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:34.199738 (+ 2301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:55895: BlockingRecv error: recv got EOF from 127.25.240.65:55895 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
Sep 24 13:38:34 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:34 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:34.313989 3274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:34.309208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:54679)
0924 13:38:34.309407 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:34.309418 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:34.309498 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:34.309679 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:34.309681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:34.309910 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:34.309968 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:34.309971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.310763 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.310764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:34.311290 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:34.311292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.311610 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.311611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:34.311650 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:34.312120 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:34.312125 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:34.313891 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":102,"threads_started":1}
W20260924 13:38:34.314069 3273 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:34.309302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:54679 (local address 127.25.240.126:40193)
0924 13:38:34.309394 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:34.309397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:34.309477 (+ 80us) server_negotiation.cc:407] Connection header received
0924 13:38:34.309590 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:34.309591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:34.309604 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:34.309671 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:34.310056 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.310691 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.311358 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.311539 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.311589 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:34.313990 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:54679: BlockingRecv error: recv got EOF from 127.25.240.66:54679 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 13:38:34 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:34 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:34.455456 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:34.451049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:44803)
0924 13:38:34.451212 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:34.451237 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:34.451303 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:34.451444 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:34.451444 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:34.451607 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:34.451653 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:34.451655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.452285 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.452287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:34.452812 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:34.452813 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.453153 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.453153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:34.453194 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:34.453696 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:34.453700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:34.455394 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
W20260924 13:38:34.455593 3273 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:34.451127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:44803 (local address 127.25.240.126:40193)
0924 13:38:34.451194 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:34.451197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:34.451300 (+ 103us) server_negotiation.cc:407] Connection header received
0924 13:38:34.451369 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:34.451370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:34.451381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:34.451429 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:34.451710 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.452199 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.452886 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:34.453063 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:34.453141 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:34.455501 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:44803: BlockingRecv error: recv got EOF from 127.25.240.67:44803 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 13:38:35 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:35 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:35.204721 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:35.200242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:60789)
0924 13:38:35.200391 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:35.200406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:35.200473 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:35.200606 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:35.200607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:35.200771 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:35.200818 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:35.200819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.201445 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.201446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:35.201992 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:35.201993 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.202309 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.202309 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:35.202355 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:35.202837 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:35.202842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:35.204652 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
W20260924 13:38:35.204847 3277 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:35.200301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:60789 (local address 127.25.240.126:40193)
0924 13:38:35.200460 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:35.200462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:35.200471 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:35.200530 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:35.200531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:35.200541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:35.200591 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:35.200883 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.201383 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.202056 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.202247 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.202300 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:35.204743 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:60789: BlockingRecv error: recv got EOF from 127.25.240.65:60789 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
Sep 24 13:38:35 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:35 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:35.318778 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:35.314384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:46021)
0924 13:38:35.314528 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:35.314539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:35.314590 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:35.314714 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:35.314715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:35.314914 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:35.314969 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:35.314971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.315637 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.315638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:35.316160 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:35.316162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.316474 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.316474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:35.316512 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:35.316996 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:35.317000 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:35.318721 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260924 13:38:35.318912 3277 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:35.314499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:46021 (local address 127.25.240.126:40193)
0924 13:38:35.314545 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:35.314546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:35.314576 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:38:35.314638 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:35.314638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:35.314647 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:35.314697 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:35.315027 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.315574 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.316226 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.316416 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.316465 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:35.318829 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:46021: BlockingRecv error: recv got EOF from 127.25.240.66:46021 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 13:38:35 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:35 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:35.460506 3279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:35.456034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:45807)
0924 13:38:35.456192 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:35.456203 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:35.456262 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:35.456400 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:35.456401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:35.456568 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:35.456612 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:35.456614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.457213 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.457214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:35.457813 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:35.457815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.458164 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.458165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:35.458214 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:35.458726 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:35.458730 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:35.460398 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
W20260924 13:38:35.460546 3277 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:35.456084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:45807 (local address 127.25.240.126:40193)
0924 13:38:35.456153 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:35.456155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:35.456262 (+ 107us) server_negotiation.cc:407] Connection header received
0924 13:38:35.456321 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:35.456321 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:35.456334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:35.456385 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:35.456671 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.457153 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.457889 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:35.458077 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:35.458153 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:35.460479 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:45807: BlockingRecv error: recv got EOF from 127.25.240.67:45807 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 13:38:36 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:36 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:36.209713 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:36.205247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:56135)
0924 13:38:36.205379 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:36.205397 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:36.205445 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:36.205611 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:36.205612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:36.205784 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:36.205832 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:36.205833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.206485 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.206486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:36.207006 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:36.207007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.207320 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.207321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:36.207359 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:36.207838 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:36.207851 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:36.209654 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
W20260924 13:38:36.209808 3281 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:36.205306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:56135 (local address 127.25.240.126:40193)
0924 13:38:36.205451 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:36.205453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:36.205462 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:36.205484 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:36.205485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:36.205494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:36.205578 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:36.205918 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.206419 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.207073 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.207260 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.207310 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:36.209728 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:56135: BlockingRecv error: recv got EOF from 127.25.240.65:56135 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
Sep 24 13:38:36 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:36 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:36.324065 3282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:36.319202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:42033)
0924 13:38:36.319382 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:36.319394 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:36.319475 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:36.319651 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:36.319652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:36.319839 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:36.319896 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:36.319897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.320558 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.320559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:36.321104 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:36.321106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.321634 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.321636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:36.321726 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:36.322181 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:36.322185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:36.324009 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
W20260924 13:38:36.324203 3281 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:36.319286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:42033 (local address 127.25.240.126:40193)
0924 13:38:36.319354 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:36.319356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:36.319454 (+ 98us) server_negotiation.cc:407] Connection header received
0924 13:38:36.319557 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:36.319558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:36.319570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:36.319641 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:36.319971 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.320477 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.321233 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.321506 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.321633 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:36.324121 (+ 2488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:42033: BlockingRecv error: recv got EOF from 127.25.240.66:42033 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 13:38:36 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:36 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.67@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:36.465642 3283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:36.461004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.67:50305)
0924 13:38:36.461156 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:36.461175 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:36.461233 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:36.461369 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:36.461370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:36.461584 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:36.461637 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:36.461639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.462412 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.462413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:36.463001 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:36.463003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.463336 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.463337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:36.463382 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:36.463794 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:36.463799 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:36.465536 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260924 13:38:36.465776 3281 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:36.461095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:50305 (local address 127.25.240.126:40193)
0924 13:38:36.461161 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:36.461162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:36.461233 (+ 71us) server_negotiation.cc:407] Connection header received
0924 13:38:36.461294 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:36.461295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:36.461306 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:36.461350 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:36.461696 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.462332 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.463067 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:36.463247 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:36.463301 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:36.465683 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.67:50305: BlockingRecv error: recv got EOF from 127.25.240.67:50305 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 13:38:37 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:37 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.65@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:37.214905 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.210283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.65:59121)
0924 13:38:37.210438 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:37.210451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:37.210511 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:37.210644 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:37.210644 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:37.210807 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:37.210854 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.210856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.211477 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.211478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.212013 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.212014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.212324 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.212325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.212363 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.212791 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:37.212795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:37.214796 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
W20260924 13:38:37.214959 3285 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:37.210337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:59121 (local address 127.25.240.126:40193)
0924 13:38:37.210491 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:37.210493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:37.210502 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:37.210562 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:37.210563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:37.210573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:37.210632 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:37.210925 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.211414 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.212084 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.212267 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.212315 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.214872 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.65:59121: BlockingRecv error: recv got EOF from 127.25.240.65:59121 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
Sep 24 13:38:37 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 13:38:37 dist-test-slave-m5z1 krb5kdc[2227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.240.66@KRBTEST.COM for oryx/127.25.240.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 13:38:37.328981 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.324624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:40193 (local address 127.25.240.66:51173)
0924 13:38:37.324785 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:37.324795 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:37.324858 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:37.325007 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:37.325008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:37.325161 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:37.325210 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.325212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.325892 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.325893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.326401 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.326402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.326709 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.326710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.326746 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.327184 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:37.327188 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:37.328923 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.240.126:40193: Server oryx/127.25.240.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
W20260924 13:38:37.329103 3285 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:37.324692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:51173 (local address 127.25.240.126:40193)
0924 13:38:37.324758 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:37.324760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:37.324854 (+ 94us) server_negotiation.cc:407] Connection header received
0924 13:38:37.324918 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:37.324919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:37.324930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:37.324980 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:37.325267 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.325826 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.326467 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.326653 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.326703 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.329018 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.240.66:51173: BlockingRecv error: recv got EOF from 127.25.240.66:51173 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:38:37.394560 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 2810
I20260924 13:38:37.399130 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 2962
I20260924 13:38:37.403751 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 3094
I20260924 13:38:37.407718 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 2740
2026-09-24T13:38:37Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17492 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T13:38:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:37Z Disabled control of system clock
I20260924 13:38:37.434834 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:44793
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:44793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 13:38:37.515583 3290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:37.515800 3290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:37.515820 3290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:37.517014 3290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:37.517057 3290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:37.517071 3290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:37.517083 3290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:37.517133 3290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:37.518569 3290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:44793
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:37.518819 3290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:37.519104 3290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:37.524062 3296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:37.524078 3295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:37.524091 3290 server_base.cc:1034] running on GCE node
W20260924 13:38:37.524082 3298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:37.524500 3290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:37.524806 3290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:37.525976 3290 hybrid_clock.cc:648] HybridClock initialized: now 1790257117525962 us; error 41 us; skew 500 ppm
I20260924 13:38:37.527140 3290 webserver.cc:460] Webserver started at http://127.25.240.126:38391/ using document root <none> and password file <none>
I20260924 13:38:37.527393 3290 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:37.527462 3290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:37.527606 3290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:37.528663 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "6167eba45ea74f0c91fe05c22ed229ca"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "08b3691528edbff48f1292f66e6bdf0a"
server_key_iv: "682a40e7a56ff944bbba36b577e7c030"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.529023 3290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "6167eba45ea74f0c91fe05c22ed229ca"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "08b3691528edbff48f1292f66e6bdf0a"
server_key_iv: "682a40e7a56ff944bbba36b577e7c030"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.530361 3290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:37.531035 3304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.531200 3290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:37.531288 3290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "6167eba45ea74f0c91fe05c22ed229ca"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "08b3691528edbff48f1292f66e6bdf0a"
server_key_iv: "682a40e7a56ff944bbba36b577e7c030"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.531363 3290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:37.547571 3290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:37.547863 3290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:37.548012 3290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:37.552004 3290 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:44793
I20260924 13:38:37.552035 3356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:44793 every 8 connection(s)
I20260924 13:38:37.552371 3290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:37.553038 3357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:37.555686 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca: Bootstrap starting.
I20260924 13:38:37.556259 3357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:37.556557 3357 log.cc:824] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:37.557220 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca: No bootstrap required, opened a new log
I20260924 13:38:37.558815 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 3290
I20260924 13:38:37.558785 3357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167eba45ea74f0c91fe05c22ed229ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44793 } }
I20260924 13:38:37.558917 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:37.558944 3357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:37.558988 3357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6167eba45ea74f0c91fe05c22ed229ca, State: Initialized, Role: FOLLOWER
I20260924 13:38:37.559145 26561 external_mini_cluster.cc:1428] Setting key 2299433f02c795dea538b8dc4441f520
I20260924 13:38:37.559144 3357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167eba45ea74f0c91fe05c22ed229ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44793 } }
I20260924 13:38:37.559222 3357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:37.559268 3357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:37.559319 3357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:37.560032 3357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167eba45ea74f0c91fe05c22ed229ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44793 } }
I20260924 13:38:37.560195 3357 leader_election.cc:302] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6167eba45ea74f0c91fe05c22ed229ca; no voters:
I20260924 13:38:37.560355 3357 leader_election.cc:288] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:37.560387 3362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:37.560559 3362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [term 1 LEADER]: Becoming Leader. State: Replica: 6167eba45ea74f0c91fe05c22ed229ca, State: Running, Role: LEADER
I20260924 13:38:37.560642 3362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167eba45ea74f0c91fe05c22ed229ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44793 } }
I20260924 13:38:37.560689 3357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:37.560956 3362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6167eba45ea74f0c91fe05c22ed229ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167eba45ea74f0c91fe05c22ed229ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44793 } } }
I20260924 13:38:37.561105 3362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:37.561318 3362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6167eba45ea74f0c91fe05c22ed229ca. Latest consensus state: current_term: 1 leader_uuid: "6167eba45ea74f0c91fe05c22ed229ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6167eba45ea74f0c91fe05c22ed229ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44793 } } }
I20260924 13:38:37.561441 3362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca [sys.catalog]: This master's current role is: LEADER
W20260924 13:38:37.562602 3378 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:38:37.562659 3378 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:38:37.562723 3369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:37.562892 3369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:37.564407 3369 catalog_manager.cc:1347] Generated new cluster ID: 1868933f00b94d13a457b61e6fc2da0a
I20260924 13:38:37.564465 3369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:37.565079 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.559649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44100 (local address 127.25.240.126:44793)
0924 13:38:37.559859 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:37.559873 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:37.559900 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:38:37.559989 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:37.559994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:37.560071 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:37.560208 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:37.560430 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.562044 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.563066 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.563505 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.563618 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.564378 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:37.564383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:37.564383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:37.564576 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:37.564617 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:37.564632 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:37.564700 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:37.564794 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 13:38:37.564878 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260924 13:38:37.583462 3369 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:37.584223 3369 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:37.594985 3369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6167eba45ea74f0c91fe05c22ed229ca: Generated new TSK 0
I20260924 13:38:37.595176 3369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:38:37.597273 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44793
--builtin_ntp_servers=127.25.240.84:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 13:38:37.678017 3381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:37.678242 3381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:37.678278 3381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:37.679430 3381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:37.679487 3381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:37.679529 3381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:37.680917 3381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:37.681171 3381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:37.681478 3381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:37.686372 3387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:37.686360 3386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:37.686378 3389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:37.686615 3381 server_base.cc:1034] running on GCE node
I20260924 13:38:37.686862 3381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:37.687112 3381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:37.688267 3381 hybrid_clock.cc:648] HybridClock initialized: now 1790257117688256 us; error 29 us; skew 500 ppm
I20260924 13:38:37.689421 3381 webserver.cc:460] Webserver started at http://127.25.240.65:40233/ using document root <none> and password file <none>
I20260924 13:38:37.689709 3381 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:37.689778 3381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:37.689922 3381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:37.690958 3381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "1c9c5b26d9f94748b2c070513be8e9fb"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "c06dc271cbc6d580adc85eef113b5383"
server_key_iv: "363809fbb872d8e95674ee202b3210eb"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.691319 3381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "1c9c5b26d9f94748b2c070513be8e9fb"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "c06dc271cbc6d580adc85eef113b5383"
server_key_iv: "363809fbb872d8e95674ee202b3210eb"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.692615 3381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:37.693315 3395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.693487 3381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:37.693622 3381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "1c9c5b26d9f94748b2c070513be8e9fb"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "c06dc271cbc6d580adc85eef113b5383"
server_key_iv: "363809fbb872d8e95674ee202b3210eb"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.693693 3381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:37.719849 3381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:37.720187 3381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:37.720343 3381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:37.720580 3381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:37.720937 3381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:37.720994 3381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.721030 3381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:37.721056 3381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.726832 3381 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:39057
I20260924 13:38:37.726877 3508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:39057 every 8 connection(s)
I20260924 13:38:37.727201 3381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:38:37.732086 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 3381
I20260924 13:38:37.732182 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:37.732440 26561 external_mini_cluster.cc:1428] Setting key ea47e85be1ecffaa87e274c53b1179a9
I20260924 13:38:37.732571 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.727953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:50341 (local address 127.25.240.126:44793)
0924 13:38:37.727997 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:37.728000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:37.728217 (+ 217us) server_negotiation.cc:407] Connection header received
0924 13:38:37.728807 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:37.728808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:37.728823 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:37.728884 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:37.729575 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.730087 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.731072 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.731294 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.731378 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.732222 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:37.732224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:37.732225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:37.732230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:37.732247 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:37.732252 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:37.732282 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:37.732426 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 13:38:37.732458 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:38:37.732838 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.727925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:44793 (local address 127.25.240.65:50341)
0924 13:38:37.728094 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:37.728131 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:37.728703 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:37.728972 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:37.728976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:37.728986 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:37.729464 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.729469 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.730183 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.730184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.730976 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.730979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.731388 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.731389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.731529 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.732037 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:37.732038 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:37.732080 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:37.732107 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:37.732331 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:37.732336 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:37.732339 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:37.732410 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:38:37.732522 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260924 13:38:37.733290 3509 heartbeater.cc:347] Connected to a master server at 127.25.240.126:44793
I20260924 13:38:37.733402 3509 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:37.733484 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44793
--builtin_ntp_servers=127.25.240.84:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 13:38:37.733628 3509 heartbeater.cc:511] Master 127.25.240.126:44793 requested a full tablet report, sending...
I20260924 13:38:37.734156 3321 ts_manager.cc:194] Registered new tserver with Master: 1c9c5b26d9f94748b2c070513be8e9fb (127.25.240.65:39057)
I20260924 13:38:37.734766 3321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.65:50341
W20260924 13:38:37.812667 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:37.813041 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:37.813086 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:37.814728 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:37.814776 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:37.814841 3512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:37.816720 3512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:37.816973 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:37.817265 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:37.822196 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:37.822238 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:37.822327 3512 server_base.cc:1034] running on GCE node
W20260924 13:38:37.822192 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:37.822577 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:37.822811 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:37.823949 3512 hybrid_clock.cc:648] HybridClock initialized: now 1790257117823933 us; error 36 us; skew 500 ppm
I20260924 13:38:37.825124 3512 webserver.cc:460] Webserver started at http://127.25.240.66:41199/ using document root <none> and password file <none>
I20260924 13:38:37.825348 3512 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:37.825393 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:37.825592 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:37.826622 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "2e409500f52146acba0249aae0e2ae58"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "39fc5747d8a416a7aa46e66766b731e1"
server_key_iv: "da29a113e84fad247699abd98811db50"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.826984 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "2e409500f52146acba0249aae0e2ae58"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "39fc5747d8a416a7aa46e66766b731e1"
server_key_iv: "da29a113e84fad247699abd98811db50"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.828233 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:37.828894 3526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.829058 3512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:37.829156 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "2e409500f52146acba0249aae0e2ae58"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "39fc5747d8a416a7aa46e66766b731e1"
server_key_iv: "da29a113e84fad247699abd98811db50"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.829226 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:37.861127 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:37.861505 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:37.861680 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:37.861919 3512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:37.862260 3512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:37.862310 3512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.862353 3512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:37.862381 3512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.867765 3512 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:35137
I20260924 13:38:37.867806 3639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:35137 every 8 connection(s)
I20260924 13:38:37.868145 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:38:37.868639 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 3512
I20260924 13:38:37.868737 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:37.868970 26561 external_mini_cluster.cc:1428] Setting key 13d67d6df28e3c8d806ccc4d4c9d1bcb
I20260924 13:38:37.869932 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44793
--builtin_ntp_servers=127.25.240.84:35103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 13:38:37.873629 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.869177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:54145 (local address 127.25.240.126:44793)
0924 13:38:37.869229 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:37.869232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:37.869245 (+ 13us) server_negotiation.cc:407] Connection header received
0924 13:38:37.869878 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:37.869879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:37.869891 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:37.869936 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:37.870683 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.871185 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.872153 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.872376 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.872450 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.873302 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:37.873304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:37.873304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:37.873309 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:37.873325 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:37.873329 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:37.873412 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:37.873501 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 13:38:37.873530 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:38:37.873819 3642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.868897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:44793 (local address 127.25.240.66:54145)
0924 13:38:37.869111 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:37.869168 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:37.869776 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:37.870011 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:37.870015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:37.870027 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:37.870573 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.870578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.871277 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.871279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.872060 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.872062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.872456 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.872457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.872589 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.873164 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:37.873164 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:37.873193 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:37.873215 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:37.873416 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:37.873421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:37.873424 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:37.873503 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:38:37.873621 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
I20260924 13:38:37.874238 3640 heartbeater.cc:347] Connected to a master server at 127.25.240.126:44793
I20260924 13:38:37.874325 3640 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:37.874506 3640 heartbeater.cc:511] Master 127.25.240.126:44793 requested a full tablet report, sending...
I20260924 13:38:37.874854 3321 ts_manager.cc:194] Registered new tserver with Master: 2e409500f52146acba0249aae0e2ae58 (127.25.240.66:35137)
I20260924 13:38:37.875391 3321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.66:54145
W20260924 13:38:37.949051 3643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:37.949246 3643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:37.949267 3643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:37.950507 3643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:37.950548 3643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:37.950574 3643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:37.951864 3643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:37.952049 3643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:37.952277 3643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:37.957058 3648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:37.957049 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:37.957078 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:37.957329 3643 server_base.cc:1034] running on GCE node
I20260924 13:38:37.957592 3643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:37.957903 3643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:37.959043 3643 hybrid_clock.cc:648] HybridClock initialized: now 1790257117959031 us; error 31 us; skew 500 ppm
I20260924 13:38:37.960211 3643 webserver.cc:460] Webserver started at http://127.25.240.67:38505/ using document root <none> and password file <none>
I20260924 13:38:37.960448 3643 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:37.960492 3643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:37.960685 3643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:37.961915 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "b2f0958e098d4dbcae46a189af42cfd6"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "d48940b11f2d666b92b5652365476c3c"
server_key_iv: "c2a1f072bb24329b719147138423b3e4"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.962266 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "b2f0958e098d4dbcae46a189af42cfd6"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "d48940b11f2d666b92b5652365476c3c"
server_key_iv: "c2a1f072bb24329b719147138423b3e4"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.963503 3643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:37.964181 3657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.964344 3643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:37.964437 3643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "b2f0958e098d4dbcae46a189af42cfd6"
format_stamp: "Formatted at 2026-09-24 13:38:37 on dist-test-slave-m5z1"
server_key: "d48940b11f2d666b92b5652365476c3c"
server_key_iv: "c2a1f072bb24329b719147138423b3e4"
server_key_version: "encryptionkey@0"
I20260924 13:38:37.964517 3643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:37.978775 3643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:37.979104 3643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:37.979254 3643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:37.979489 3643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:37.979822 3643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:37.979872 3643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.979923 3643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:37.979954 3643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:37.985769 3643 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:34229
I20260924 13:38:37.985813 3770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:34229 every 8 connection(s)
I20260924 13:38:37.986135 3643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:38:37.990932 3361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.986802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:58387 (local address 127.25.240.126:44793)
0924 13:38:37.986867 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:37.986869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:37.987096 (+ 227us) server_negotiation.cc:407] Connection header received
0924 13:38:37.987581 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:37.987581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:37.987590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:37.987645 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:37.988274 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.988760 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.989768 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.989946 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.989997 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.990717 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:37.990719 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:37.990719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:37.990723 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:37.990736 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:37.990739 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:37.990781 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:37.990851 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 13:38:37.990875 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:38:37.991160 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:37.986796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:44793 (local address 127.25.240.67:58387)
0924 13:38:37.986975 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:37.987014 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:37.987498 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:37.987705 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:37.987708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:37.987716 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:37.988203 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.988207 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.988836 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.988837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.989697 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:37.989699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:37.990014 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:37.990015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:37.990119 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:37.990587 (+ 468us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:37.990588 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:37.990615 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:37.990636 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:37.990789 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:37.990792 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:37.990794 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:38:37.990845 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:38:37.990921 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260924 13:38:37.991650 3771 heartbeater.cc:347] Connected to a master server at 127.25.240.126:44793
I20260924 13:38:37.991743 3771 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:37.991920 3771 heartbeater.cc:511] Master 127.25.240.126:44793 requested a full tablet report, sending...
I20260924 13:38:37.992261 3321 ts_manager.cc:194] Registered new tserver with Master: b2f0958e098d4dbcae46a189af42cfd6 (127.25.240.67:34229)
I20260924 13:38:37.992615 3321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.67:58387
I20260924 13:38:37.994740 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 3643
I20260924 13:38:37.994841 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:37.995033 26561 external_mini_cluster.cc:1428] Setting key fea36a9b35074c41b89f4f094f6d4616
I20260924 13:38:37.995797 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 13:38:37.997658 3361 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:37.997064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.25.240.126:44793)
0924 13:38:37.997100 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:37.997101 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:37.997264 (+ 163us) server_negotiation.cc:407] Connection header received
0924 13:38:37.997338 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:37.997338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:37.997352 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:37.997423 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:37.997549 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44112: BlockingRecv error: recv got EOF from 127.0.0.1:44112 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:37.997726 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 3381
I20260924 13:38:38.002838 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 3512
I20260924 13:38:38.007705 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 3643
I20260924 13:38:38.011875 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 3290
2026-09-24T13:38:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (603 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T13:38:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:38Z Disabled control of system clock
I20260924 13:38:38.037652 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:33667
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:33667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 13:38:38.116375 3782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:38.116567 3782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:38.116588 3782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:38.117650 3782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:38.117682 3782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:38.117694 3782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:38.117707 3782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:38.117719 3782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:38.118968 3782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:33667
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:33667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:38.119148 3782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:38.119362 3782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:38.124060 3788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:38.124069 3787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:38.124097 3782 server_base.cc:1034] running on GCE node
W20260924 13:38:38.124069 3790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:38.124464 3782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:38.124737 3782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:38.125895 3782 hybrid_clock.cc:648] HybridClock initialized: now 1790257118125873 us; error 47 us; skew 500 ppm
I20260924 13:38:38.127048 3782 webserver.cc:460] Webserver started at http://127.25.240.126:36153/ using document root <none> and password file <none>
I20260924 13:38:38.127270 3782 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:38.127334 3782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:38.127486 3782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:38.128512 3782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "36a34b340ad100ee7c8efaf6dc720e62"
server_key_iv: "9cd0bd6c556ab49798f26de40ddbcdf6"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.128861 3782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "36a34b340ad100ee7c8efaf6dc720e62"
server_key_iv: "9cd0bd6c556ab49798f26de40ddbcdf6"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.130200 3782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:38.130915 3796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.131093 3782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:38.131196 3782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "36a34b340ad100ee7c8efaf6dc720e62"
server_key_iv: "9cd0bd6c556ab49798f26de40ddbcdf6"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.131251 3782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:38.134420 3782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:38.134629 3782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:38.134720 3782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:38.138255 3782 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:33667
I20260924 13:38:38.138302 3848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:33667 every 8 connection(s)
I20260924 13:38:38.138597 3782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:38.139191 3849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:38.141319 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 3782
I20260924 13:38:38.141398 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:38.141605 26561 external_mini_cluster.cc:1428] Setting key 1c89611e20fb2ac456a4d0dcf6582448
I20260924 13:38:38.142079 3849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca: Bootstrap starting.
I20260924 13:38:38.142752 3849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:38.143090 3849 log.cc:824] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:38.143783 3849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca: No bootstrap required, opened a new log
I20260924 13:38:38.144345 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:38.142137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45048 (local address 127.25.240.126:33667)
0924 13:38:38.142313 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:38.142321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:38.142352 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:38:38.142971 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:38.142980 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:38.143036 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:38.143185 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:38.143918 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:38.143922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:38.143922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:38.143926 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:38.143977 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:38.143987 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:38.144077 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:38.144120 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 13:38:38.144200 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260924 13:38:38.145821 3849 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33667 } }
I20260924 13:38:38.145972 3849 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:38.146016 3849 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6f5cc5c950b4221ba785e0f2d16a9ca, State: Initialized, Role: FOLLOWER
I20260924 13:38:38.146137 3849 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33667 } }
I20260924 13:38:38.146211 3849 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:38.146255 3849 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:38.146304 3849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:38.146791 3849 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33667 } }
I20260924 13:38:38.146905 3849 leader_election.cc:302] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6f5cc5c950b4221ba785e0f2d16a9ca; no voters:
I20260924 13:38:38.147074 3849 leader_election.cc:288] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:38.147107 3854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:38.147264 3854 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [term 1 LEADER]: Becoming Leader. State: Replica: a6f5cc5c950b4221ba785e0f2d16a9ca, State: Running, Role: LEADER
I20260924 13:38:38.147334 3849 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:38.147367 3854 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33667 } }
I20260924 13:38:38.147728 3855 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33667 } } }
I20260924 13:38:38.147848 3855 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:38.147724 3856 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6f5cc5c950b4221ba785e0f2d16a9ca. Latest consensus state: current_term: 1 leader_uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6f5cc5c950b4221ba785e0f2d16a9ca" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33667 } } }
I20260924 13:38:38.147951 3856 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:38.148396 3862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:38.148808 3862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:38.150391 3862 catalog_manager.cc:1347] Generated new cluster ID: d73afed4824a49d486bb779d8174bb09
I20260924 13:38:38.150435 3862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:38.178455 3862 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:38.178650 3862 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:38.186510 3862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6f5cc5c950b4221ba785e0f2d16a9ca: Generated new TSK 0
I20260924 13:38:38.186705 3862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:38:38.191866 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33667
--builtin_ntp_servers=127.25.240.84:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 13:38:38.272250 3873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:38.272464 3873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:38.272487 3873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:38.273717 3873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:38.273756 3873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:38.273801 3873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:38.275123 3873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:38.275338 3873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:38.275619 3873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:38.280761 3881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:38.280753 3878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:38.280754 3879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:38.280961 3873 server_base.cc:1034] running on GCE node
I20260924 13:38:38.281219 3873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:38.281476 3873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:38.282632 3873 hybrid_clock.cc:648] HybridClock initialized: now 1790257118282614 us; error 29 us; skew 500 ppm
I20260924 13:38:38.291213 3873 webserver.cc:460] Webserver started at http://127.25.240.65:40739/ using document root <none> and password file <none>
I20260924 13:38:38.291474 3873 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:38.291541 3873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:38.291683 3873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:38.292778 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "9454808d54414be6894f89f27863f9bc"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "11ebc64573c3eef11834bb5b522541a3"
server_key_iv: "0ad9310bbdf630176e26833b0671b857"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.293143 3873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "9454808d54414be6894f89f27863f9bc"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "11ebc64573c3eef11834bb5b522541a3"
server_key_iv: "0ad9310bbdf630176e26833b0671b857"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.294529 3873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:38.295229 3887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.295389 3873 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:38.295492 3873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "9454808d54414be6894f89f27863f9bc"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "11ebc64573c3eef11834bb5b522541a3"
server_key_iv: "0ad9310bbdf630176e26833b0671b857"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.295567 3873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:38.298884 3873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:38.299136 3873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:38.299273 3873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:38.299470 3873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:38.299769 3873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:38.299819 3873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.299857 3873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:38.299887 3873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.305336 3873 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:46377
I20260924 13:38:38.305398 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:46377 every 8 connection(s)
I20260924 13:38:38.305785 3873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:38:38.306522 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 3873
I20260924 13:38:38.306619 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:38.306804 26561 external_mini_cluster.cc:1428] Setting key 3bc1ec6f59e9c4db321e9171780f6b89
I20260924 13:38:38.307806 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33667
--builtin_ntp_servers=127.25.240.84:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 13:38:38.309545 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:38.306510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:37313 (local address 127.25.240.126:33667)
0924 13:38:38.306587 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:38.306590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:38.307169 (+ 579us) server_negotiation.cc:407] Connection header received
0924 13:38:38.307819 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:38.307820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:38.307825 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:38.307897 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:38.309272 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:38.309274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:38.309274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:38.309280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:38.309299 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:38.309304 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:38.309369 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:38.309440 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 13:38:38.309457 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:38:38.309751 4003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:38.306482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33667 (local address 127.25.240.65:37313)
0924 13:38:38.307037 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:38.307090 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:38.307705 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:38.307977 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:38.307982 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:38.307991 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:38.309077 (+ 1086us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:38.309078 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:38.309120 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:38.309160 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:38.309374 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:38.309379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:38.309382 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:38.309442 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 13:38:38.309491 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":93,"threads_started":1}
I20260924 13:38:38.310180 4001 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33667
I20260924 13:38:38.310276 4001 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:38.310423 4001 heartbeater.cc:511] Master 127.25.240.126:33667 requested a full tablet report, sending...
I20260924 13:38:38.310877 3813 ts_manager.cc:194] Registered new tserver with Master: 9454808d54414be6894f89f27863f9bc (127.25.240.65:46377)
W20260924 13:38:38.388756 4004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:38.388955 4004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:38.388978 4004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:38.390209 4004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:38.390252 4004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:38.390296 4004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:38.391626 4004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:38.391855 4004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:38.392131 4004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:38:38.397073 4004 server_base.cc:1034] running on GCE node
W20260924 13:38:38.397099 4009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:38.397197 4012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:38.397089 4010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:38.397462 4004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:38.397765 4004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:38.398929 4004 hybrid_clock.cc:648] HybridClock initialized: now 1790257118398923 us; error 33 us; skew 500 ppm
I20260924 13:38:38.400156 4004 webserver.cc:460] Webserver started at http://127.25.240.66:41945/ using document root <none> and password file <none>
I20260924 13:38:38.400374 4004 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:38.400420 4004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:38.400568 4004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:38.401672 4004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "706749fcd50f4216b5c5580816c22fc0"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "31f0fa2fa02809329cc661b56a848713"
server_key_iv: "fefd7e070cb34285a5e150d4e9b12f99"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.402022 4004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "706749fcd50f4216b5c5580816c22fc0"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "31f0fa2fa02809329cc661b56a848713"
server_key_iv: "fefd7e070cb34285a5e150d4e9b12f99"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.403232 4004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:38:38.403885 4018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.404048 4004 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:38.404129 4004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "706749fcd50f4216b5c5580816c22fc0"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "31f0fa2fa02809329cc661b56a848713"
server_key_iv: "fefd7e070cb34285a5e150d4e9b12f99"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.404197 4004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:38.407430 4004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:38.407667 4004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:38.407790 4004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:38.407991 4004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:38.408279 4004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:38.408310 4004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.408360 4004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:38.408388 4004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.413774 4004 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:40815
I20260924 13:38:38.413823 4131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:40815 every 8 connection(s)
I20260924 13:38:38.414103 4004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:38:38.416750 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:38.414767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:37729 (local address 127.25.240.126:33667)
0924 13:38:38.414845 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:38.414847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:38.415013 (+ 166us) server_negotiation.cc:407] Connection header received
0924 13:38:38.415420 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:38.415421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:38.415427 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:38.415478 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:38.416569 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:38.416570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:38.416570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:38.416572 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:38.416582 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:38.416585 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:38.416624 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:38.416670 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 13:38:38.416681 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:38:38.416952 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:38.414735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33667 (local address 127.25.240.66:37729)
0924 13:38:38.414906 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:38.414937 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:38.415344 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:38.415528 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:38.415530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:38.415538 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:38.416448 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:38.416449 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:38.416476 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:38.416502 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:38.416624 (+ 122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:38.416627 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:38.416629 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:38:38.416679 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:38:38.416732 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":91,"threads_started":1}
I20260924 13:38:38.417418 4132 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33667
I20260924 13:38:38.417507 4132 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:38.417680 4132 heartbeater.cc:511] Master 127.25.240.126:33667 requested a full tablet report, sending...
I20260924 13:38:38.418025 3813 ts_manager.cc:194] Registered new tserver with Master: 706749fcd50f4216b5c5580816c22fc0 (127.25.240.66:40815)
I20260924 13:38:38.422889 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 4004
I20260924 13:38:38.423020 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:38.423276 26561 external_mini_cluster.cc:1428] Setting key 1bdad0058a022318b6ec4b9f40aead39
I20260924 13:38:38.424268 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33667
--builtin_ntp_servers=127.25.240.84:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 13:38:38.508675 4135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:38.508862 4135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:38.508882 4135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:38.510113 4135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:38.510151 4135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:38.510176 4135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:38.511598 4135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:38.511849 4135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:38.512121 4135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:38.516920 4143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:38.516937 4140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:38.517184 4135 server_base.cc:1034] running on GCE node
W20260924 13:38:38.516930 4141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:38.517423 4135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:38.517695 4135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:38.518846 4135 hybrid_clock.cc:648] HybridClock initialized: now 1790257118518842 us; error 32 us; skew 500 ppm
I20260924 13:38:38.519986 4135 webserver.cc:460] Webserver started at http://127.25.240.67:33333/ using document root <none> and password file <none>
I20260924 13:38:38.520222 4135 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:38.520285 4135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:38.520423 4135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:38.521494 4135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "776c372e6e6b4f05a3724aa22a91d548"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "504d3600fd15124db8965e96ad66f255"
server_key_iv: "07f4303e8ddbc0f517706e17f28861b8"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.521931 4135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "776c372e6e6b4f05a3724aa22a91d548"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "504d3600fd15124db8965e96ad66f255"
server_key_iv: "07f4303e8ddbc0f517706e17f28861b8"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.523203 4135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:38:38.523859 4149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.524024 4135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:38.524106 4135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "776c372e6e6b4f05a3724aa22a91d548"
format_stamp: "Formatted at 2026-09-24 13:38:38 on dist-test-slave-m5z1"
server_key: "504d3600fd15124db8965e96ad66f255"
server_key_iv: "07f4303e8ddbc0f517706e17f28861b8"
server_key_version: "encryptionkey@0"
I20260924 13:38:38.524180 4135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:38.527410 4135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:38.527668 4135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:38.527799 4135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:38.527997 4135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:38.528294 4135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:38.528344 4135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.528383 4135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:38.528411 4135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:38.534003 4135 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:44159
I20260924 13:38:38.534044 4262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:44159 every 8 connection(s)
I20260924 13:38:38.534324 4135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:38:38.536844 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:38.534952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:52855 (local address 127.25.240.126:33667)
0924 13:38:38.535004 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:38.535006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:38.535222 (+ 216us) server_negotiation.cc:407] Connection header received
0924 13:38:38.535623 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:38.535624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:38.535629 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:38.535679 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:38.536668 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:38.536669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:38.536669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:38:38.536673 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:38.536687 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:38:38.536691 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:38.536731 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:38.536782 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 13:38:38.536793 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:38:38.537014 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:38.534960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33667 (local address 127.25.240.67:52855)
0924 13:38:38.535117 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:38.535147 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:38.535547 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:38.535727 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:38.535730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:38.535738 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:38.536547 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:38:38.536548 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:38.536571 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:38:38.536595 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:38.536736 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:38.536738 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:38.536740 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:38:38.536787 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:38:38.536828 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260924 13:38:38.537451 4263 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33667
I20260924 13:38:38.537544 4263 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:38.537709 4263 heartbeater.cc:511] Master 127.25.240.126:33667 requested a full tablet report, sending...
I20260924 13:38:38.538071 3813 ts_manager.cc:194] Registered new tserver with Master: 776c372e6e6b4f05a3724aa22a91d548 (127.25.240.67:44159)
I20260924 13:38:38.542707 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 4135
I20260924 13:38:38.542796 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:38.542984 26561 external_mini_cluster.cc:1428] Setting key 7a671c2ad73f386792bc74bc874cd87f
I20260924 13:38:38.543805 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 13:38:38.545550 3852 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:38:38.545067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.25.240.126:33667)
0924 13:38:38.545105 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:38.545106 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:38.545237 (+ 131us) server_negotiation.cc:407] Connection header received
0924 13:38:38.545311 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:38.545312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:38.545316 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:38.545371 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:38.545483 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45056: BlockingRecv error: recv got EOF from 127.0.0.1:45056 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:38.545603 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 3873
I20260924 13:38:38.550323 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 4004
I20260924 13:38:38.555248 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 4135
I20260924 13:38:38.559334 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 3782
2026-09-24T13:38:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (547 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:38 dist-test-slave-m5z1 krb5kdc[4273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:38 dist-test-slave-m5z1 krb5kdc[4273](info): set up 2 sockets
Sep 24 13:38:38 dist-test-slave-m5z1 krb5kdc[4273](info): commencing operation
krb5kdc: starting...
W20260924 13:38:39.586858 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:39 dist-test-slave-m5z1 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:38:39.708302 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:41857
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:41857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:39.788082 4286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:39.788324 4286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:39.788359 4286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:39.789511 4286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:38:39.789603 4286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:39.789640 4286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:39.789665 4286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:38:39.789693 4286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:38:39.791148 4286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:41857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:41857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:39.791412 4286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:39.791630 4286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:39.796378 4291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:39.796423 4286 server_base.cc:1034] running on GCE node
W20260924 13:38:39.796393 4292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:39.796393 4294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:39.796792 4286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:39.797073 4286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:39.798235 4286 hybrid_clock.cc:648] HybridClock initialized: now 1790257119798222 us; error 47 us; skew 500 ppm
Sep 24 13:38:39 dist-test-slave-m5z1 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257119, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:39.800544 4286 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:38:39.800958 4286 webserver.cc:460] Webserver started at http://127.25.240.126:41871/ using document root <none> and password file <none>
I20260924 13:38:39.801187 4286 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:39.801250 4286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:39.801390 4286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:39.802475 4286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "1fb6a178fd064b7b9b2bb485aac0a83e"
format_stamp: "Formatted at 2026-09-24 13:38:39 on dist-test-slave-m5z1"
server_key: "ea28f50fe8c4688135042791eca985ce"
server_key_iv: "6188041a699c81d03a8a859994b13a2d"
server_key_version: "encryptionkey@0"
I20260924 13:38:39.802825 4286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "1fb6a178fd064b7b9b2bb485aac0a83e"
format_stamp: "Formatted at 2026-09-24 13:38:39 on dist-test-slave-m5z1"
server_key: "ea28f50fe8c4688135042791eca985ce"
server_key_iv: "6188041a699c81d03a8a859994b13a2d"
server_key_version: "encryptionkey@0"
I20260924 13:38:39.804081 4286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:38:39.804749 4301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:39.804912 4286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:39.804998 4286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "1fb6a178fd064b7b9b2bb485aac0a83e"
format_stamp: "Formatted at 2026-09-24 13:38:39 on dist-test-slave-m5z1"
server_key: "ea28f50fe8c4688135042791eca985ce"
server_key_iv: "6188041a699c81d03a8a859994b13a2d"
server_key_version: "encryptionkey@0"
I20260924 13:38:39.805078 4286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:39.813967 4286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:39.815593 4286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:39.815742 4286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:39.819470 4286 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:41857
I20260924 13:38:39.819500 4353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:41857 every 8 connection(s)
I20260924 13:38:39.819844 4286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:38:39.821166 4354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:39.822784 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 4286
I20260924 13:38:39.822887 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:38:39.823098 26561 external_mini_cluster.cc:1428] Setting key c002df25c2ee42ab1f2e0dbbc683afe4
I20260924 13:38:39.824404 4354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e: Bootstrap starting.
I20260924 13:38:39.825160 4354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:39.825464 4354 log.cc:824] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:39.826171 4354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e: No bootstrap required, opened a new log
I20260924 13:38:39.828581 4354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb6a178fd064b7b9b2bb485aac0a83e" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41857 } }
I20260924 13:38:39.828760 4354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:39.828799 4354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb6a178fd064b7b9b2bb485aac0a83e, State: Initialized, Role: FOLLOWER
I20260924 13:38:39.828943 4354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb6a178fd064b7b9b2bb485aac0a83e" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41857 } }
I20260924 13:38:39.829052 4354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:38:39.829103 4354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:38:39.829161 4354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:39.829873 4354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb6a178fd064b7b9b2bb485aac0a83e" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41857 } }
I20260924 13:38:39.829986 4354 leader_election.cc:302] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fb6a178fd064b7b9b2bb485aac0a83e; no voters:
Sep 24 13:38:39 dist-test-slave-m5z1 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:39.830139 4354 leader_election.cc:288] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:38:39.830175 4359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:39.830320 4359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [term 1 LEADER]: Becoming Leader. State: Replica: 1fb6a178fd064b7b9b2bb485aac0a83e, State: Running, Role: LEADER
I20260924 13:38:39.830396 4354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:38:39.830430 4359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb6a178fd064b7b9b2bb485aac0a83e" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41857 } }
I20260924 13:38:39.830756 4360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fb6a178fd064b7b9b2bb485aac0a83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb6a178fd064b7b9b2bb485aac0a83e" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41857 } } }
I20260924 13:38:39.830845 4360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:39.831132 4361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fb6a178fd064b7b9b2bb485aac0a83e. Latest consensus state: current_term: 1 leader_uuid: "1fb6a178fd064b7b9b2bb485aac0a83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fb6a178fd064b7b9b2bb485aac0a83e" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41857 } } }
I20260924 13:38:39.831246 4361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e [sys.catalog]: This master's current role is: LEADER
I20260924 13:38:39.831408 4366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:38:39.832038 4366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:38:39.833897 4366 catalog_manager.cc:1347] Generated new cluster ID: 3eb5c0b2e50f4dff85c2f12b1d2660ea
I20260924 13:38:39.833956 4366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:38:39.834426 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:39.823602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.25.240.126:41857)
0924 13:38:39.823768 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:39.823776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:39.823812 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:38:39.824216 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:39.824222 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:39.824265 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:39.824376 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:39.825182 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:39.826351 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:39.827388 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:39.827767 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:39.827843 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:39.830617 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:39.830622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:39.830622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:39.830634 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:39.832758 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:39.833163 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:39.833166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:39.833166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:39.833265 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:39.833488 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:39.833490 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:39.833490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:39.833806 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:39.833981 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:39.834099 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 13:38:39.834237 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260924 13:38:39.843631 4366 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:38:39.844111 4366 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:38:39.853348 4366 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e: Generated new TSK 0
I20260924 13:38:39.853531 4366 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:38:39.886454 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:41857
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:39.965750 4380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:39.965955 4380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:39.965981 4380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:39.967136 4380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:39.967177 4380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:39.967203 4380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:38:39.968623 4380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:39.968849 4380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:39.969092 4380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:39.974186 4388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:39.974335 4380 server_base.cc:1034] running on GCE node
W20260924 13:38:39.974133 4385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:39.974134 4386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:39.974639 4380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:39.974892 4380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:39.976063 4380 hybrid_clock.cc:648] HybridClock initialized: now 1790257119976048 us; error 31 us; skew 500 ppm
Sep 24 13:38:39 dist-test-slave-m5z1 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257119, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:39.978701 4380 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:38:39.979251 4380 webserver.cc:460] Webserver started at http://127.25.240.65:42315/ using document root <none> and password file <none>
I20260924 13:38:39.979521 4380 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:39.979590 4380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:39.979728 4380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:39.980762 4380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "08573957f6ef48b8b13cfd9561e44ab8"
format_stamp: "Formatted at 2026-09-24 13:38:39 on dist-test-slave-m5z1"
server_key: "2847918305fc5f232c01c39aca45987c"
server_key_iv: "076568bd73f4a7b4e2ba245ab91353be"
server_key_version: "encryptionkey@0"
I20260924 13:38:39.981115 4380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "08573957f6ef48b8b13cfd9561e44ab8"
format_stamp: "Formatted at 2026-09-24 13:38:39 on dist-test-slave-m5z1"
server_key: "2847918305fc5f232c01c39aca45987c"
server_key_iv: "076568bd73f4a7b4e2ba245ab91353be"
server_key_version: "encryptionkey@0"
I20260924 13:38:39.982509 4380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:39.983263 4395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:39.983445 4380 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:39.983543 4380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "08573957f6ef48b8b13cfd9561e44ab8"
format_stamp: "Formatted at 2026-09-24 13:38:39 on dist-test-slave-m5z1"
server_key: "2847918305fc5f232c01c39aca45987c"
server_key_iv: "076568bd73f4a7b4e2ba245ab91353be"
server_key_version: "encryptionkey@0"
I20260924 13:38:39.983625 4380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:39.994360 4380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:39.996642 4380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:39.996805 4380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:39.997023 4380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:39.997318 4380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:39.997370 4380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:39.997411 4380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:39.997437 4380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:40.003249 4380 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:40025
I20260924 13:38:40.003290 4508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:40025 every 8 connection(s)
I20260924 13:38:40.003592 4380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:38:40 dist-test-slave-m5z1 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257119, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:40.011040 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 4380
I20260924 13:38:40.011148 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:38:40.011373 26561 external_mini_cluster.cc:1428] Setting key 026dbba92fd67509062be9b0e06fb256
I20260924 13:38:40.012390 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.004230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:41857 (local address 127.25.240.65:41135)
0924 13:38:40.004368 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.004404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:40.004870 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:40.005062 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:40.005065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:40.005697 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:40.005893 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.005898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.006575 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.006576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.007370 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.007372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.007785 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.007786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.007883 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.008282 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:40.008289 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:40.009641 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:40.011022 (+ 1381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:40.011030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:40.011033 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:40.011331 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:40.011535 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:40.011536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:40.011536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:40.011647 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:40.011896 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:40.011902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:40.012036 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 13:38:40.012122 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 13:38:40.012236 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260924 13:38:40.012390 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.004252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:41135 (local address 127.25.240.126:41857)
0924 13:38:40.004297 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.004299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.004494 (+ 195us) server_negotiation.cc:407] Connection header received
0924 13:38:40.004948 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.004950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.004959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.005012 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:40.005977 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.006497 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.007452 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.007716 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.007776 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.009744 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:40.009745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:40.009746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:40.009752 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:40.010938 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:40.011408 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:40.011409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:40.011409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:40.011487 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:40.011708 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:40.011709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:40.011709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:40.011832 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:40.011883 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.012148 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.012253 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:38:40.013226 4509 heartbeater.cc:347] Connected to a master server at 127.25.240.126:41857
I20260924 13:38:40.013334 4509 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:40.013514 4509 heartbeater.cc:511] Master 127.25.240.126:41857 requested a full tablet report, sending...
I20260924 13:38:40.014048 4318 ts_manager.cc:194] Registered new tserver with Master: 08573957f6ef48b8b13cfd9561e44ab8 (127.25.240.65:40025)
I20260924 13:38:40.014632 4318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:41135
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:38:40.035725 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:41857
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:40.116066 4514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:40.116266 4514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:40.116287 4514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:40.117448 4514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:40.117489 4514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:40.117539 4514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:38:40.118993 4514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:40.119230 4514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:40.119501 4514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:40.124550 4522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:40.124567 4520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:40.124559 4519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:40.124830 4514 server_base.cc:1034] running on GCE node
I20260924 13:38:40.125063 4514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:40.125258 4514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:40.126386 4514 hybrid_clock.cc:648] HybridClock initialized: now 1790257120126375 us; error 28 us; skew 500 ppm
Sep 24 13:38:40 dist-test-slave-m5z1 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257120, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:40.128706 4514 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:38:40.129168 4514 webserver.cc:460] Webserver started at http://127.25.240.66:41121/ using document root <none> and password file <none>
I20260924 13:38:40.129383 4514 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:40.129448 4514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:40.129635 4514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:40.130767 4514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "d52d344b464e4e78bcef555b57421d2f"
format_stamp: "Formatted at 2026-09-24 13:38:40 on dist-test-slave-m5z1"
server_key: "d442857bf273ac6cc1c9165335ae9ab3"
server_key_iv: "f2b4f0abe3862ba15a042299ace10200"
server_key_version: "encryptionkey@0"
I20260924 13:38:40.131124 4514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "d52d344b464e4e78bcef555b57421d2f"
format_stamp: "Formatted at 2026-09-24 13:38:40 on dist-test-slave-m5z1"
server_key: "d442857bf273ac6cc1c9165335ae9ab3"
server_key_iv: "f2b4f0abe3862ba15a042299ace10200"
server_key_version: "encryptionkey@0"
I20260924 13:38:40.132493 4514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:38:40.133162 4529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:40.133332 4514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:38:40.133428 4514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "d52d344b464e4e78bcef555b57421d2f"
format_stamp: "Formatted at 2026-09-24 13:38:40 on dist-test-slave-m5z1"
server_key: "d442857bf273ac6cc1c9165335ae9ab3"
server_key_iv: "f2b4f0abe3862ba15a042299ace10200"
server_key_version: "encryptionkey@0"
I20260924 13:38:40.133497 4514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:40.164803 4514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:40.166965 4514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:40.167128 4514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:40.167358 4514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:40.167670 4514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:40.167719 4514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:40.167759 4514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:40.167789 4514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:40.173599 4514 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:35543
I20260924 13:38:40.173629 4642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:35543 every 8 connection(s)
I20260924 13:38:40.173945 4514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:38:40 dist-test-slave-m5z1 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257120, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:40.180647 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 4514
I20260924 13:38:40.180768 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:38:40.180985 26561 external_mini_cluster.cc:1428] Setting key fe68af51d8598646ebe33c791f84b099
I20260924 13:38:40.183733 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.174624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:35559 (local address 127.25.240.126:41857)
0924 13:38:40.174691 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.174693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.174909 (+ 216us) server_negotiation.cc:407] Connection header received
0924 13:38:40.175447 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.175449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.175461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.175517 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:40.176432 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.176941 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.177990 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.178210 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.178293 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.180436 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:40.180437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:40.180437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:40.180445 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:40.182406 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:40.182761 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:40.182762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:40.182763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:40.182827 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:40.183044 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:40.183044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:40.183045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:40.183205 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:40.183280 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.183605 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.183670 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:38:40.184013 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.174620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:41857 (local address 127.25.240.66:35559)
0924 13:38:40.174793 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.174829 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:40.175324 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:40.175586 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:40.175590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:40.176138 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:40.176328 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.176333 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.177035 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.177037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.177894 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.177897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.178301 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.178302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.178427 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.178906 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:40.178924 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:40.180335 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:40.182483 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:40.182488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:40.182490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:40.182692 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:40.182889 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:40.182889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:40.182890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:40.182987 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:40.183279 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:40.183286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:40.183464 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 13:38:40.183586 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 13:38:40.183761 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260924 13:38:40.184510 4643 heartbeater.cc:347] Connected to a master server at 127.25.240.126:41857
I20260924 13:38:40.184603 4643 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:40.184770 4643 heartbeater.cc:511] Master 127.25.240.126:41857 requested a full tablet report, sending...
I20260924 13:38:40.185138 4318 ts_manager.cc:194] Registered new tserver with Master: d52d344b464e4e78bcef555b57421d2f (127.25.240.66:35543)
I20260924 13:38:40.185708 4318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:35559
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:38:40.208325 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:41857
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:38:40.288972 4648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:38:40.289172 4648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:38:40.289196 4648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:38:40.290396 4648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:38:40.290441 4648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:38:40.290493 4648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:38:40.291949 4648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:41857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:38:40.292191 4648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:38:40.292418 4648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:38:40.297248 4653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:40.297291 4656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:38:40.297478 4654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:38:40.297508 4648 server_base.cc:1034] running on GCE node
I20260924 13:38:40.297722 4648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:38:40.297961 4648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:38:40.299117 4648 hybrid_clock.cc:648] HybridClock initialized: now 1790257120299109 us; error 34 us; skew 500 ppm
Sep 24 13:38:40 dist-test-slave-m5z1 krb5kdc[4273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257120, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:38:40.301462 4648 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:38:40.301942 4648 webserver.cc:460] Webserver started at http://127.25.240.67:36031/ using document root <none> and password file <none>
I20260924 13:38:40.302171 4648 fs_manager.cc:359] Metadata directory not provided
I20260924 13:38:40.302233 4648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:38:40.302376 4648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:38:40.303395 4648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "b4e790aed5d94c849bf6ccd16621b2cc"
format_stamp: "Formatted at 2026-09-24 13:38:40 on dist-test-slave-m5z1"
server_key: "d461d3d9aca09ef50feb9b3fe5b7be78"
server_key_iv: "731e207b08c9fd13dd061222df5f35ae"
server_key_version: "encryptionkey@0"
I20260924 13:38:40.303743 4648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "b4e790aed5d94c849bf6ccd16621b2cc"
format_stamp: "Formatted at 2026-09-24 13:38:40 on dist-test-slave-m5z1"
server_key: "d461d3d9aca09ef50feb9b3fe5b7be78"
server_key_iv: "731e207b08c9fd13dd061222df5f35ae"
server_key_version: "encryptionkey@0"
I20260924 13:38:40.305043 4648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:38:40.305742 4663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:40.305910 4648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:38:40.306010 4648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "b4e790aed5d94c849bf6ccd16621b2cc"
format_stamp: "Formatted at 2026-09-24 13:38:40 on dist-test-slave-m5z1"
server_key: "d461d3d9aca09ef50feb9b3fe5b7be78"
server_key_iv: "731e207b08c9fd13dd061222df5f35ae"
server_key_version: "encryptionkey@0"
I20260924 13:38:40.306090 4648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:38:40.324446 4648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:38:40.326701 4648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:38:40.326855 4648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:38:40.327086 4648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:38:40.327381 4648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:38:40.327430 4648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:40.327466 4648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:38:40.327490 4648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:38:40.333528 4648 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:42915
I20260924 13:38:40.333626 4776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:42915 every 8 connection(s)
I20260924 13:38:40.333920 4648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:38:40 dist-test-slave-m5z1 krb5kdc[4273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257120, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:38:40.342762 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.334647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:52067 (local address 127.25.240.126:41857)
0924 13:38:40.334695 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.334698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.334912 (+ 214us) server_negotiation.cc:407] Connection header received
0924 13:38:40.335363 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.335364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.335374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.335426 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:40.336357 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.336871 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.337956 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.338165 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.338250 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.340220 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:40.340222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:40.340222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:40.340229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:40.341362 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:40.341823 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:40.341825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:40.341825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:40.341885 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:40.342135 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:40.342136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:40.342136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:40.342250 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:40.342316 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.342583 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.342668 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:38:40.342957 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.334636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:41857 (local address 127.25.240.67:52067)
0924 13:38:40.334795 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.334830 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:40.335286 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:40.335479 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:40.335483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:40.336056 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:38:40.336245 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.336249 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.336968 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.336969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.337844 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.337846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.338261 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.338261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.338384 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.338807 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:38:40.338815 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:38:40.340125 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:38:40.341466 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:40.341472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:40.341474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:40.341750 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:40.341952 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:38:40.341953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:38:40.341953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:38:40.342062 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:38:40.342317 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:38:40.342324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:38:40.342458 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 13:38:40.342563 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 13:38:40.342726 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260924 13:38:40.343391 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 4648
I20260924 13:38:40.343434 4777 heartbeater.cc:347] Connected to a master server at 127.25.240.126:41857
I20260924 13:38:40.343505 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:38:40.343554 4777 heartbeater.cc:464] Registering TS with master...
I20260924 13:38:40.343698 26561 external_mini_cluster.cc:1428] Setting key fe4bf9f3868ab4df25c1b115cf9d9452
I20260924 13:38:40.343742 4777 heartbeater.cc:511] Master 127.25.240.126:41857 requested a full tablet report, sending...
I20260924 13:38:40.344189 4318 ts_manager.cc:194] Registered new tserver with Master: b4e790aed5d94c849bf6ccd16621b2cc (127.25.240.67:42915)
I20260924 13:38:40.344594 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:38:40.344635 4318 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:52067
I20260924 13:38:40.351279 4357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.346043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.25.240.126:41857)
0924 13:38:40.346102 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.346104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.346190 (+ 86us) server_negotiation.cc:407] Connection header received
0924 13:38:40.346244 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.346244 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.346252 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.346295 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:38:40.346804 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.347244 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.347897 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.348068 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.348115 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.349224 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:38:40.349225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:38:40.349225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:38:40.349230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:38:40.350390 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:40.350685 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:40.350686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:40.350686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:40.350730 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:38:40.350881 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:38:40.350882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:38:40.350882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:38:40.351002 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:38:40.351077 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.351150 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.351203 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 13:38:40.352671 4318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:38:40.354557 4318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:38:40.360956 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.357032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.25.240.66:35543)
0924 13:38:40.357222 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.357225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.357233 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:40.357282 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.357284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.357299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.357550 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:40.357645 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.358431 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.359865 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.360540 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.360609 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.360655 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.360828 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.360873 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260924 13:38:40.361135 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.357312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.25.240.65:40025)
0924 13:38:40.357519 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.357523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.357532 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:38:40.357601 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.357603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.357617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.357692 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:40.357887 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.358914 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.360108 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.360623 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.360692 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.360751 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.361017 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.361063 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260924 13:38:40.361003 4789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.356712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:35543 (local address 127.0.0.1:49322)
0924 13:38:40.356855 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.356867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:40.356929 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:40.357405 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:40.357406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:40.357409 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:40.357471 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.357472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.358529 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.358530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.359776 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.359779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.360623 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.360624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.360715 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.360728 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:38:40.360802 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 13:38:40.360859 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":54,"thread_start_us":74,"threads_started":1}
I20260924 13:38:40.361232 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.356809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.25.240.67:42915)
0924 13:38:40.356980 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.356985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.356995 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:40.357032 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.357034 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.357047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.357116 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:40.357312 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.358453 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.359694 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.360492 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.360567 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.360634 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.361098 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.361139 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260924 13:38:40.360976 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.356698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:42915 (local address 127.0.0.1:47412)
0924 13:38:40.356799 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.356839 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:40.356897 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:40.357145 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:40.357149 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:40.357151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:40.357216 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.357218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.358568 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.358569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.359605 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.359608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.360673 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.360674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.360738 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.360742 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:40.360822 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:38:40.360868 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":25,"thread_start_us":58,"threads_started":1}
I20260924 13:38:40.361140 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.356897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40025 (local address 127.0.0.1:53764)
0924 13:38:40.357303 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.357313 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:40.357357 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:40.357768 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:40.357769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:40.357771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:40.357822 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.357824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.359021 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.359022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.360026 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.360028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.360784 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.360786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.360847 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.360851 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:40.360896 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 13:38:40.360934 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":69,"threads_started":1}
I20260924 13:38:40.361967 4577 tablet_service.cc:1467] Processing CreateTablet for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6 (DEFAULT_TABLE table=test-table [id=9d9ddaddb47847da957c629de015fc49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:40.361964 4711 tablet_service.cc:1467] Processing CreateTablet for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6 (DEFAULT_TABLE table=test-table [id=9d9ddaddb47847da957c629de015fc49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:40.362068 4443 tablet_service.cc:1467] Processing CreateTablet for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6 (DEFAULT_TABLE table=test-table [id=9d9ddaddb47847da957c629de015fc49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:38:40.362362 4577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:40.362362 4711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:40.362470 4443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:38:40.365553 4794 tablet_bootstrap.cc:492] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8: Bootstrap starting.
I20260924 13:38:40.365875 4795 tablet_bootstrap.cc:492] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: Bootstrap starting.
I20260924 13:38:40.366400 4796 tablet_bootstrap.cc:492] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: Bootstrap starting.
I20260924 13:38:40.366492 4794 tablet_bootstrap.cc:654] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:40.366526 4795 tablet_bootstrap.cc:654] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:40.366976 4794 log.cc:824] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:40.367072 4796 tablet_bootstrap.cc:654] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: Neither blocks nor log segments found. Creating new log.
I20260924 13:38:40.367118 4795 log.cc:824] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:40.367645 4796 log.cc:824] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: Log is configured to *not* fsync() on all Append() calls
I20260924 13:38:40.367835 4795 tablet_bootstrap.cc:492] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: No bootstrap required, opened a new log
I20260924 13:38:40.367861 4794 tablet_bootstrap.cc:492] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8: No bootstrap required, opened a new log
I20260924 13:38:40.367913 4795 ts_tablet_manager.cc:1397] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:40.367919 4794 ts_tablet_manager.cc:1397] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:40.368386 4796 tablet_bootstrap.cc:492] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: No bootstrap required, opened a new log
I20260924 13:38:40.368515 4796 ts_tablet_manager.cc:1397] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:40.369422 4795 raft_consensus.cc:348] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.369593 4795 raft_consensus.cc:374] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:40.369643 4795 raft_consensus.cc:729] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52d344b464e4e78bcef555b57421d2f, State: Initialized, Role: FOLLOWER
I20260924 13:38:40.369760 4795 consensus_queue.cc:260] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.370026 4795 ts_tablet_manager.cc:1428] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:38:40.369940 4794 raft_consensus.cc:348] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.370095 4794 raft_consensus.cc:374] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:40.370141 4794 raft_consensus.cc:729] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08573957f6ef48b8b13cfd9561e44ab8, State: Initialized, Role: FOLLOWER
I20260924 13:38:40.370251 4794 consensus_queue.cc:260] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.370523 4509 heartbeater.cc:503] Master 127.25.240.126:41857 was elected leader, sending a full tablet report...
I20260924 13:38:40.370586 4643 heartbeater.cc:503] Master 127.25.240.126:41857 was elected leader, sending a full tablet report...
I20260924 13:38:40.370654 4796 raft_consensus.cc:348] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.370793 4796 raft_consensus.cc:374] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:38:40.370829 4796 raft_consensus.cc:729] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e790aed5d94c849bf6ccd16621b2cc, State: Initialized, Role: FOLLOWER
I20260924 13:38:40.370951 4796 consensus_queue.cc:260] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.371218 4796 ts_tablet_manager.cc:1428] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 13:38:40.371227 4777 heartbeater.cc:503] Master 127.25.240.126:41857 was elected leader, sending a full tablet report...
I20260924 13:38:40.370522 4794 ts_tablet_manager.cc:1428] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:38:40.383539 4801 raft_consensus.cc:482] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:38:40.383651 4801 raft_consensus.cc:504] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.384042 4801 leader_election.cc:288] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4e790aed5d94c849bf6ccd16621b2cc (127.25.240.67:42915), d52d344b464e4e78bcef555b57421d2f (127.25.240.66:35543)
I20260924 13:38:40.387091 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.384254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:37365 (local address 127.25.240.67:42915)
0924 13:38:40.384351 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.384353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.384361 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:38:40.384456 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.384457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.384466 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.384519 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:40.384676 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.385172 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.386310 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.386755 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.386830 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.386840 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.386996 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.387030 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 13:38:40.387091 4511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.384156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:42915 (local address 127.25.240.65:37365)
0924 13:38:40.384314 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.384327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:40.384374 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:40.384527 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:40.384528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:40.384530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:40.384606 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.384609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.385256 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.385257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.386205 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.386207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.386837 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.386838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.386918 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.386922 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:38:40.386979 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:38:40.387020 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":54}
I20260924 13:38:40.387143 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.384255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:47325 (local address 127.25.240.66:35543)
0924 13:38:40.384314 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.384317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.384424 (+ 107us) server_negotiation.cc:407] Connection header received
0924 13:38:40.384525 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.384526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.384537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.384594 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:38:40.384801 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.385390 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.386336 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.386800 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.386888 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.386898 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.387041 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.387077 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:38:40.387202 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.384201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:35543 (local address 127.25.240.65:47325)
0924 13:38:40.384360 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.384371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:38:40.384446 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:38:40.384605 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:38:40.384606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:38:40.384607 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:38:40.384683 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.384684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.385470 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.385471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.386241 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:38:40.386243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.386888 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.386889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:38:40.386962 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.386965 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:38:40.387034 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 13:38:40.387074 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260924 13:38:40.387486 4597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307c7cf8c8ca4f5d9b45faeba9ee63f6" candidate_uuid: "08573957f6ef48b8b13cfd9561e44ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d52d344b464e4e78bcef555b57421d2f" is_pre_election: true
I20260924 13:38:40.387650 4597 raft_consensus.cc:2454] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08573957f6ef48b8b13cfd9561e44ab8 in term 0.
I20260924 13:38:40.387709 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307c7cf8c8ca4f5d9b45faeba9ee63f6" candidate_uuid: "08573957f6ef48b8b13cfd9561e44ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" is_pre_election: true
I20260924 13:38:40.387877 4731 raft_consensus.cc:2454] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08573957f6ef48b8b13cfd9561e44ab8 in term 0.
I20260924 13:38:40.387867 4397 leader_election.cc:302] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08573957f6ef48b8b13cfd9561e44ab8, d52d344b464e4e78bcef555b57421d2f; no voters:
I20260924 13:38:40.387998 4801 raft_consensus.cc:2781] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:38:40.388054 4801 raft_consensus.cc:482] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:38:40.388078 4801 raft_consensus.cc:3037] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:40.388797 4801 raft_consensus.cc:504] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.388998 4801 leader_election.cc:288] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [CANDIDATE]: Term 1 election: Requested vote from peers b4e790aed5d94c849bf6ccd16621b2cc (127.25.240.67:42915), d52d344b464e4e78bcef555b57421d2f (127.25.240.66:35543)
I20260924 13:38:40.389063 4597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307c7cf8c8ca4f5d9b45faeba9ee63f6" candidate_uuid: "08573957f6ef48b8b13cfd9561e44ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d52d344b464e4e78bcef555b57421d2f"
I20260924 13:38:40.389079 4731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307c7cf8c8ca4f5d9b45faeba9ee63f6" candidate_uuid: "08573957f6ef48b8b13cfd9561e44ab8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4e790aed5d94c849bf6ccd16621b2cc"
I20260924 13:38:40.389148 4597 raft_consensus.cc:3037] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:40.389145 4731 raft_consensus.cc:3037] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:38:40.389896 4731 raft_consensus.cc:2454] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08573957f6ef48b8b13cfd9561e44ab8 in term 1.
I20260924 13:38:40.389966 4597 raft_consensus.cc:2454] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08573957f6ef48b8b13cfd9561e44ab8 in term 1.
I20260924 13:38:40.390060 4396 leader_election.cc:302] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08573957f6ef48b8b13cfd9561e44ab8, b4e790aed5d94c849bf6ccd16621b2cc; no voters:
I20260924 13:38:40.390158 4801 raft_consensus.cc:2781] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:38:40.390314 4801 raft_consensus.cc:686] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [term 1 LEADER]: Becoming Leader. State: Replica: 08573957f6ef48b8b13cfd9561e44ab8, State: Running, Role: LEADER
I20260924 13:38:40.390404 4801 consensus_queue.cc:237] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } }
I20260924 13:38:40.391261 4316 catalog_manager.cc:5495] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08573957f6ef48b8b13cfd9561e44ab8 (127.25.240.65). New cstate: current_term: 1 leader_uuid: "08573957f6ef48b8b13cfd9561e44ab8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08573957f6ef48b8b13cfd9561e44ab8" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 40025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 } health_report { overall_health: UNKNOWN } } }
I20260924 13:38:40.396057 4793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.393793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.25.240.65:40025)
0924 13:38:40.393836 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:38:40.393838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:38:40.393848 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:38:40.393922 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:38:40.393922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:38:40.393940 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:38:40.394002 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:38:40.394104 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.394552 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.395299 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:38:40.395502 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:38:40.395574 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:38:40.395736 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:38:40.395838 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:38:40.395928 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:38:40.395947 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 13:38:40.395966 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:38:40.397657 4731 raft_consensus.cc:1261] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 1 FOLLOWER]: Refusing update from remote peer 08573957f6ef48b8b13cfd9561e44ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:40.397655 4597 raft_consensus.cc:1261] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 1 FOLLOWER]: Refusing update from remote peer 08573957f6ef48b8b13cfd9561e44ab8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:38:40.397987 4804 consensus_queue.cc:1035] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4e790aed5d94c849bf6ccd16621b2cc" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:40.398103 4801 consensus_queue.cc:1035] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P 08573957f6ef48b8b13cfd9561e44ab8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d52d344b464e4e78bcef555b57421d2f" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:38:40.400817 4808 mvcc.cc:205] Tried to move back new op lower bound from 7332893165146337280 to 7332893165119721472. Current Snapshot: MvccSnapshot[applied={T|T < 7332893165146337280}]
I20260924 13:38:40.402262 4316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:40.402362 4316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:38:40.403074 4316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1fb6a178fd064b7b9b2bb485aac0a83e: Sending DeleteTablet for 3 replicas of tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6
I20260924 13:38:40.403385 4577 tablet_service.cc:1514] Processing DeleteTablet for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:40 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:49322
I20260924 13:38:40.403448 4443 tablet_service.cc:1514] Processing DeleteTablet for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:40 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:53764
I20260924 13:38:40.403434 4711 tablet_service.cc:1514] Processing DeleteTablet for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:38:40 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:47412
I20260924 13:38:40.403861 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 4380
I20260924 13:38:40.404503 4816 tablet_replica.cc:330] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: stopping tablet replica
I20260924 13:38:40.404611 4816 raft_consensus.cc:2229] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:40.404716 4816 raft_consensus.cc:2258] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:40.404906 4818 tablet_replica.cc:330] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: stopping tablet replica
I20260924 13:38:40.404999 4818 raft_consensus.cc:2229] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:38:40.405090 4818 raft_consensus.cc:2258] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:38:40.405100 4816 ts_tablet_manager.cc:1905] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:40.405457 4818 ts_tablet_manager.cc:1905] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:38:40.406775 4816 ts_tablet_manager.cc:1918] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:40.406862 4816 log.cc:1196] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/307c7cf8c8ca4f5d9b45faeba9ee63f6
I20260924 13:38:40.407182 4816 ts_tablet_manager.cc:1939] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P d52d344b464e4e78bcef555b57421d2f: Deleting consensus metadata
I20260924 13:38:40.407433 4818 ts_tablet_manager.cc:1918] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:38:40.407511 4818 log.cc:1196] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/307c7cf8c8ca4f5d9b45faeba9ee63f6
I20260924 13:38:40.407621 4303 catalog_manager.cc:4841] TS d52d344b464e4e78bcef555b57421d2f (127.25.240.66:35543): tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6 (table test-table [id=9d9ddaddb47847da957c629de015fc49]) successfully deleted
I20260924 13:38:40.407778 4818 ts_tablet_manager.cc:1939] T 307c7cf8c8ca4f5d9b45faeba9ee63f6 P b4e790aed5d94c849bf6ccd16621b2cc: Deleting consensus metadata
I20260924 13:38:40.408233 4302 catalog_manager.cc:4841] TS b4e790aed5d94c849bf6ccd16621b2cc (127.25.240.67:42915): tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6 (table test-table [id=9d9ddaddb47847da957c629de015fc49]) successfully deleted
W20260924 13:38:40.411062 4302 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:40025 (error 108)
I20260924 13:38:40.411324 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 4514
I20260924 13:38:40.411892 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:38:40.411411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:40025 (local address 127.0.0.1:53772)
0924 13:38:40.411814 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 13:38:40.411853 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:40025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382}
W20260924 13:38:40.412317 4302 catalog_manager.cc:4584] TS 08573957f6ef48b8b13cfd9561e44ab8 (127.25.240.65:40025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 307c7cf8c8ca4f5d9b45faeba9ee63f6: Network error: Client connection negotiation failed: client connection to 127.25.240.65:40025: connect: Connection refused (error 111)
I20260924 13:38:40.416236 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 4648
I20260924 13:38:40.420727 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 4286
2026-09-24T13:38:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1862 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:38:40 dist-test-slave-m5z1 krb5kdc[4822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:38:40 dist-test-slave-m5z1 krb5kdc[4822](info): set up 2 sockets
Sep 24 13:38:40 dist-test-slave-m5z1 krb5kdc[4822](info): commencing operation
krb5kdc: starting...
W20260924 13:38:41.482821 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:38:41 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:38:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:38:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/rangeradmin_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/rangeradmin_127.25.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/rangerlookup_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/rangerlookup_127.25.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/HTTP_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/HTTP_127.25.240.84@KRBTEST.COM.keytab.
I20260924 13:38:41.641753 26561 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFHYdCJ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/postgres -l logfile start
2026-09-24 13:38:44.148 UTC [4852] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 13:38:44.148 UTC [4852] LOG: listening on IPv4 address "127.25.240.84", port 33839
2026-09-24 13:38:44.152 UTC [4852] LOG: listening on Unix socket "/tmp/.s.PGSQL.33839"
2026-09-24 13:38:44.187 UTC [4855] LOG: database system was shut down at 2026-09-24 13:38:42 UTC
2026-09-24 13:38:44.192 UTC [4852] LOG: database system is ready to accept connections
I20260924 13:38:45.138751 26561 mini_postgres.cc:96] Postgres bound to 33839
2026-09-24 13:38:45.144 UTC [4863] FATAL: database "slave" does not exist
127.25.240.84:33839 - accepting connections
I20260924 13:38:45.145550 26561 mini_ranger.cc:161] Starting Ranger...
I20260924 13:38:45.161216 26561 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 13:38:45.652346 26561 mini_ranger.cc:87] Created ranger Postgres database
I20260924 13:38:45.652433 26561 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-admin
2026-09-24 13:38:45,924 [I] DB FLAVOR :POSTGRES
2026-09-24 13:38:45,925 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:38:45,925 [I] Checking connection..
2026-09-24 13:38:45,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 13:38:46,156 [I] Checking connection passed.
2026-09-24 13:38:46,156 [I] --------- Verifying version history table ---------
2026-09-24 13:38:46,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:38:46,416 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:38:46,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:38:46,662 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:38:46,662 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 13:38:46,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 13:38:46.931 UTC [4927] WARNING: there is no transaction in progress
2026-09-24 13:38:46,937 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 13:38:46,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:38:47,191 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 13:38:47,191 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:38:47,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 13:38:47,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 13:38:47,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m5z1','N') ;"
2026-09-24 13:38:47,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:38:48,192 [I] Table x_portal_user does not exist in database ranger
2026-09-24 13:38:48,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:38:48,445 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 13:38:48,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:38:48,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:38:48,976 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 13:38:48,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 13:38:49.927 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:49.942 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:49.959 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:49.974 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:49.991 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.011 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.071 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.077 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.092 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.102 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.112 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.125 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.134 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.140 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.146 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50.152 UTC [5035] WARNING: there is no transaction in progress
2026-09-24 13:38:50,805 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 13:38:50,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 13:38:51,054 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 13:38:51,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:38:51,304 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 13:38:51,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:38:51,530 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 13:38:51,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:38:51,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:38:52,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m5z1';"
2026-09-24 13:38:52,265 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 13:38:52,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 13:38:52,506 [I] DB_PATCHES have already been applied
I20260924 13:38:52.511672 26561 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFHYdCJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 13:38:52.511734 26561 mini_ranger.cc:193] Using host: 127.25.240.84
I20260924 13:38:52.514689 26561 mini_ranger.cc:227] Ranger admin URL: http://127.25.240.84:37129
Sep 24, 2026 1:38:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:38:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:38:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 1:38:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 1:38:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 1:38:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 1:38:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 13:38:53.604912 26561 mini_ranger.cc:160] Time spent starting Ranger: real 8.459s user 0.003s sys 0.006s
Sep 24, 2026 1:39:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 1:39:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 1:39:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 1:39:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 1:39:13 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 13:39:14.291250 26561 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/rangerkms_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/rangerkms_127.25.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.240.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.240.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.240.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/HTTP_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.240.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/HTTP_127.25.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 13:39:14 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257154, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 13:39:14.374713 26561 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 13:39:14.384068 26561 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 13:39:15.270896 26561 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 13:39:15.270987 26561 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms
I20260924 13:39:15.271008 26561 mini_ranger_kms.cc:227] Using postgres at 127.25.240.84:33839
2026-09-24 13:39:15,447 [I] DB FLAVOR :POSTGRES
2026-09-24 13:39:15,447 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:39:15,447 [I] Checking connection
2026-09-24 13:39:15,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 13:39:15,694 [I] connection success
2026-09-24 13:39:15,694 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 13:39:15,695 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 13:39:15,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 13:39:15,920 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 13:39:15,921 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:39:15,921 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 13:39:15,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:33839/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 13:39:16,168 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 13:39:16.639729 26561 mini_ranger_kms.cc:310] Created kms service
I20260924 13:39:16.824364 26561 mini_ranger_kms.cc:326] Created kudu user
I20260924 13:39:16.871735 26561 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 13:39:16 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257154, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:39:17.274536 26561 mini_ranger_kms.cc:384] Added ranger policy
I20260924 13:39:17.274659 26561 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 13:39:17.274680 26561 mini_ranger_kms.cc:240] Using host: 127.25.240.84
I20260924 13:39:17.278122 26561 mini_ranger_kms.cc:276] Ranger KMS PID: 5215
I20260924 13:39:17.278245 26561 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.240.84:38395
Sep 24, 2026 1:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 1:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 1:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 1:39:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 13:39:19.481328 26561 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.107s user 0.000s sys 0.158s
I20260924 13:39:19.481446 26561 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 13:39:19.481523 26561 mini_ranger_kms.cc:401] 127.25.240.84:38395/kms/v1/keys
Sep 24 13:39:19 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.240.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 13:39:20 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.240.84@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
Sep 24 13:39:20 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257159, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.240.84@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
Sep 24 13:39:22 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:39:22.532419 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:34325
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:34325
--ranger_config_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:38395/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:39:22.612279 5266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:39:22.612489 5266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:39:22.612510 5266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:39:22.613675 5266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:39:22.613716 5266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:39:22.613731 5266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:39:22.613744 5266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:39:22.613755 5266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:39:22.615320 5266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.240.84:38395/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:34325
--ranger_config_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:34325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:39:22.615556 5266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:39:22.615800 5266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:39:22.620999 5274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:39:22.621100 5271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:39:22.621009 5272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:39:22.621006 5266 server_base.cc:1034] running on GCE node
I20260924 13:39:22.621508 5266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:39:22.621851 5266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:39:22.623040 5266 hybrid_clock.cc:648] HybridClock initialized: now 1790257162623022 us; error 33 us; skew 500 ppm
Sep 24 13:39:22 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257162, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:39:22.631939 5266 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:39:22.632479 5266 webserver.cc:460] Webserver started at http://127.25.240.126:36237/ using document root <none> and password file <none>
I20260924 13:39:22.632741 5266 fs_manager.cc:359] Metadata directory not provided
I20260924 13:39:22.632795 5266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:39:22.632930 5266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:39:22 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257162, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:39:22.686160 5266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "9f99420dff814186b65c0c1e5b9d7469"
format_stamp: "Formatted at 2026-09-24 13:39:22 on dist-test-slave-m5z1"
server_key: "8b7abe7a9811ccc172f3309f084e1455"
server_key_iv: "e4614aba7edce35902b47f1a1fed6b99"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:22.686591 5266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "9f99420dff814186b65c0c1e5b9d7469"
format_stamp: "Formatted at 2026-09-24 13:39:22 on dist-test-slave-m5z1"
server_key: "8b7abe7a9811ccc172f3309f084e1455"
server_key_iv: "e4614aba7edce35902b47f1a1fed6b99"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:22.688791 5266 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 13:39:22.701612 5281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:22.701876 5266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:39:22.701989 5266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "9f99420dff814186b65c0c1e5b9d7469"
format_stamp: "Formatted at 2026-09-24 13:39:22 on dist-test-slave-m5z1"
server_key: "8b7abe7a9811ccc172f3309f084e1455"
server_key_iv: "e4614aba7edce35902b47f1a1fed6b99"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:22.702087 5266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:39:22.731817 5266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:39:22.733899 5266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:39:22.734072 5266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:39:22.738921 5266 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:34325
I20260924 13:39:22.738947 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:34325 every 8 connection(s)
I20260924 13:39:22.739363 5266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:39:22.740094 5334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:39:22.742494 5334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469: Bootstrap starting.
I20260924 13:39:22.743257 5334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469: Neither blocks nor log segments found. Creating new log.
I20260924 13:39:22.743595 5334 log.cc:824] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469: Log is configured to *not* fsync() on all Append() calls
I20260924 13:39:22.744369 5334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469: No bootstrap required, opened a new log
I20260924 13:39:22.746536 5334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99420dff814186b65c0c1e5b9d7469" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34325 } }
I20260924 13:39:22.746685 5334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:39:22.746707 5334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f99420dff814186b65c0c1e5b9d7469, State: Initialized, Role: FOLLOWER
I20260924 13:39:22.746872 5334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99420dff814186b65c0c1e5b9d7469" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34325 } }
I20260924 13:39:22.746945 5334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:39:22.746968 5334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:39:22.747031 5334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:39:22.747568 5334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99420dff814186b65c0c1e5b9d7469" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34325 } }
I20260924 13:39:22.747653 5334 leader_election.cc:302] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f99420dff814186b65c0c1e5b9d7469; no voters:
I20260924 13:39:22.747867 5334 leader_election.cc:288] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:39:22.747921 5337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:39:22.748162 5337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [term 1 LEADER]: Becoming Leader. State: Replica: 9f99420dff814186b65c0c1e5b9d7469, State: Running, Role: LEADER
I20260924 13:39:22.748211 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 5266
I20260924 13:39:22.748301 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:39:22.748298 5337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99420dff814186b65c0c1e5b9d7469" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34325 } }
I20260924 13:39:22.748519 26561 external_mini_cluster.cc:1428] Setting key a1509450b23be6eb58d91ab522643e7f
I20260924 13:39:22.748620 5337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f99420dff814186b65c0c1e5b9d7469. Latest consensus state: current_term: 1 leader_uuid: "9f99420dff814186b65c0c1e5b9d7469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99420dff814186b65c0c1e5b9d7469" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34325 } } }
I20260924 13:39:22.748688 5337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [sys.catalog]: This master's current role is: LEADER
I20260924 13:39:22.748972 5334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:39:22.748632 5339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f99420dff814186b65c0c1e5b9d7469" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f99420dff814186b65c0c1e5b9d7469" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 34325 } } }
I20260924 13:39:22.749172 5339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469 [sys.catalog]: This master's current role is: LEADER
I20260924 13:39:22.750313 5341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:39:22.751012 5341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:39:22.751430 5334 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 24 13:39:22 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:39:22.763689 5341 catalog_manager.cc:1347] Generated new cluster ID: fb8f3f531bfd49f589eaaa0d3506403f
I20260924 13:39:22.763756 5341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:39:22.765389 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:22.749159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55738 (local address 127.25.240.126:34325)
0924 13:39:22.749311 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:22.749329 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:22.749366 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:39:22.749498 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:22.749505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:22.749605 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:22.750680 (+ 1075us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:39:22.750793 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:22.755483 (+ 4690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:22.756708 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:22.757450 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:22.757933 (+ 483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:22.760863 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:39:22.760867 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:39:22.760868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:39:22.760880 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:39:22.763213 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:22.763626 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:22.763628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:22.763629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:22.763731 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:22.763974 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:22.763975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:22.763975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:22.764337 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:39:22.764950 (+ 613us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:22.765026 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 13:39:22.765163 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260924 13:39:22.784312 5341 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:39:22.785127 5341 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:39:22.789268 5341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469: Generated new TSK 0
I20260924 13:39:22.789505 5341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:39:23.528863 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:38395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34325
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:39:23.625268 5379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:39:23.625483 5379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:39:23.625520 5379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:39:23.626696 5379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:39:23.626760 5379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:39:23.626802 5379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:39:23.628235 5379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.240.84:38395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:39:23.628490 5379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:39:23.628720 5379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:39:23.633734 5384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:39:23.633734 5387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:39:23.634176 5379 server_base.cc:1034] running on GCE node
W20260924 13:39:23.634032 5385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:39:23.634558 5379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:39:23.634809 5379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:39:23.638095 5379 hybrid_clock.cc:648] HybridClock initialized: now 1790257163638071 us; error 36 us; skew 500 ppm
Sep 24 13:39:23 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257163, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:39:23.641477 5379 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:39:23.642091 5379 webserver.cc:460] Webserver started at http://127.25.240.65:46615/ using document root <none> and password file <none>
I20260924 13:39:23.642419 5379 fs_manager.cc:359] Metadata directory not provided
I20260924 13:39:23.642537 5379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:39:23.642716 5379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:39:23 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257163, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:39:23.656677 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "6521e005ac084fc4a2e58a7207a425fb"
format_stamp: "Formatted at 2026-09-24 13:39:23 on dist-test-slave-m5z1"
server_key: "c2f02218a11049e6917a994e8e023ea5"
server_key_iv: "a913916014665595e2ae7ef76257ae23"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:23.657136 5379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "6521e005ac084fc4a2e58a7207a425fb"
format_stamp: "Formatted at 2026-09-24 13:39:23 on dist-test-slave-m5z1"
server_key: "c2f02218a11049e6917a994e8e023ea5"
server_key_iv: "a913916014665595e2ae7ef76257ae23"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:23.659052 5379 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260924 13:39:23.677831 5396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:23.678094 5379 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260924 13:39:23.678248 5379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "6521e005ac084fc4a2e58a7207a425fb"
format_stamp: "Formatted at 2026-09-24 13:39:23 on dist-test-slave-m5z1"
server_key: "c2f02218a11049e6917a994e8e023ea5"
server_key_iv: "a913916014665595e2ae7ef76257ae23"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:23.678429 5379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:39:23.747661 5379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:39:23.750176 5379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:39:23.750447 5379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:39:23.750763 5379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:39:23.751174 5379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:39:23.751277 5379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:23.751382 5379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:39:23.751463 5379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:23.758713 5379 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:43993
I20260924 13:39:23.759254 5379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:39:23.760299 5511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:43993 every 8 connection(s)
I20260924 13:39:23.766160 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 5379
I20260924 13:39:23.766268 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:39:23.766497 26561 external_mini_cluster.cc:1428] Setting key e8da08328b3a63ccbb50b364a428148f
Sep 24 13:39:23 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257163, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:39:23.771756 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:23.761455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:34325 (local address 127.25.240.65:43865)
0924 13:39:23.761696 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:23.761738 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:39:23.762260 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:39:23.762543 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:39:23.762548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:39:23.762749 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:39:23.762951 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:23.762956 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:23.763678 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:23.763679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:23.764504 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:23.764506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:23.764935 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:23.764936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:23.765345 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:23.765899 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:39:23.765915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:39:23.767832 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:39:23.770027 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:39:23.770035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:39:23.770038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:39:23.770367 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:39:23.770591 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:39:23.770592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:39:23.770592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:39:23.770721 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:39:23.771027 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:39:23.771033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:39:23.771209 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 13:39:23.771329 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 13:39:23.771482 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
I20260924 13:39:23.772390 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:23.761470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:43865 (local address 127.25.240.126:34325)
0924 13:39:23.761637 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:23.761639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:23.761828 (+ 189us) server_negotiation.cc:407] Connection header received
0924 13:39:23.762367 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:23.762368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:23.762382 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:23.762460 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:39:23.763052 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:23.763584 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:23.764646 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:23.764854 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:23.764931 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:23.767946 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:39:23.767948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:39:23.767949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:39:23.767957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:39:23.769933 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:23.770455 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:23.770456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:23.770457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:23.770523 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:23.770788 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:23.770789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:23.770789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:23.770940 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:39:23.771000 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:23.772156 (+ 1156us) server_negotiation.cc:299] Negotiation successful
0924 13:39:23.772249 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260924 13:39:23.773115 5512 heartbeater.cc:347] Connected to a master server at 127.25.240.126:34325
I20260924 13:39:23.773211 5512 heartbeater.cc:464] Registering TS with master...
I20260924 13:39:23.773397 5512 heartbeater.cc:511] Master 127.25.240.126:34325 requested a full tablet report, sending...
I20260924 13:39:23.774016 5298 ts_manager.cc:194] Registered new tserver with Master: 6521e005ac084fc4a2e58a7207a425fb (127.25.240.65:43993)
I20260924 13:39:23.774649 5298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:43865
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:39:23.796396 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:38395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34325
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:39:23 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257163, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 13:39:23.893857 5518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:39:23.894098 5518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:39:23.894122 5518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:39:23.895282 5518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:39:23.895323 5518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:39:23.895349 5518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:39:23.896773 5518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.240.84:38395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:39:23.896994 5518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:39:23.897218 5518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:39:23.902283 5525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:39:23.902357 5524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:39:23.902283 5527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:39:23.903061 5518 server_base.cc:1034] running on GCE node
I20260924 13:39:23.903293 5518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:39:23.903532 5518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:39:23.906764 5518 hybrid_clock.cc:648] HybridClock initialized: now 1790257163906743 us; error 32 us; skew 500 ppm
Sep 24 13:39:23 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257163, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:39:23.909684 5518 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:39:23.910192 5518 webserver.cc:460] Webserver started at http://127.25.240.66:42715/ using document root <none> and password file <none>
I20260924 13:39:23.910439 5518 fs_manager.cc:359] Metadata directory not provided
I20260924 13:39:23.910511 5518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:39:23.910647 5518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:39:23 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257163, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:39:23.932075 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "869564eb1cdf418db751cd2feb7342f5"
format_stamp: "Formatted at 2026-09-24 13:39:23 on dist-test-slave-m5z1"
server_key: "6243e71d66c3a72e4074a4bc52e772ed"
server_key_iv: "5cd7e214a2506c7225b70753bf1f6142"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:23.932518 5518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "869564eb1cdf418db751cd2feb7342f5"
format_stamp: "Formatted at 2026-09-24 13:39:23 on dist-test-slave-m5z1"
server_key: "6243e71d66c3a72e4074a4bc52e772ed"
server_key_iv: "5cd7e214a2506c7225b70753bf1f6142"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:23.934043 5518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:39:23.946600 5534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:23.946952 5518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:39:23.947070 5518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "869564eb1cdf418db751cd2feb7342f5"
format_stamp: "Formatted at 2026-09-24 13:39:23 on dist-test-slave-m5z1"
server_key: "6243e71d66c3a72e4074a4bc52e772ed"
server_key_iv: "5cd7e214a2506c7225b70753bf1f6142"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:23.947184 5518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:39:23.970845 5518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:39:23.973171 5518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:39:23.973378 5518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:39:23.973686 5518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:39:23.974145 5518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:39:23.974234 5518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:23.974309 5518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:39:23.974366 5518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:23.981803 5518 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:44739
I20260924 13:39:23.982355 5518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:39:23.992436 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 5518
I20260924 13:39:23.992563 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:39:23.992789 26561 external_mini_cluster.cc:1428] Setting key 4869cd374ce98d046a5e8e9678cd58c7
I20260924 13:39:23.994014 5647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:44739 every 8 connection(s)
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Sep 24 13:39:24 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257163, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:39:24.016053 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:23.995133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:57141 (local address 127.25.240.126:34325)
0924 13:39:23.995542 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:23.995545 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.000761 (+ 5216us) server_negotiation.cc:407] Connection header received
0924 13:39:24.002029 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.002031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.002050 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.002080 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:39:24.002994 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.003873 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.009272 (+ 5399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.009625 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.009676 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.012583 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:39:24.012585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:39:24.012586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:39:24.012594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:39:24.014413 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:24.014929 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:24.014930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:24.014931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:24.014991 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:24.015270 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:24.015271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:24.015271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:24.015423 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:39:24.015470 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.015850 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.015944 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375}
I20260924 13:39:24.016913 5649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:23.994751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:34325 (local address 127.25.240.66:57141)
0924 13:39:24.000596 (+ 5845us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.000653 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 13:39:24.001894 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:39:24.002289 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:39:24.002294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:39:24.002578 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:39:24.002878 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.002885 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.003988 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.003989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.009151 (+ 5162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.009154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.009772 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.009773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.009920 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.010562 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:39:24.010573 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:39:24.012463 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:39:24.014512 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:39:24.014519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:39:24.014522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:39:24.014831 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:39:24.015064 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:39:24.015065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:39:24.015065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:39:24.015192 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:39:24.015522 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:39:24.015528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:39:24.015695 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 13:39:24.016490 (+ 795us) client_negotiation.cc:240] Negotiation successful
0924 13:39:24.016659 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5791,"thread_start_us":134,"threads_started":1}
I20260924 13:39:24.017671 5648 heartbeater.cc:347] Connected to a master server at 127.25.240.126:34325
I20260924 13:39:24.017767 5648 heartbeater.cc:464] Registering TS with master...
I20260924 13:39:24.017964 5648 heartbeater.cc:511] Master 127.25.240.126:34325 requested a full tablet report, sending...
I20260924 13:39:24.018440 5298 ts_manager.cc:194] Registered new tserver with Master: 869564eb1cdf418db751cd2feb7342f5 (127.25.240.66:44739)
I20260924 13:39:24.018935 5298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:57141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:39:24.044050 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:38395/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34325
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:44917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:39:24.157979 5653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:39:24.158265 5653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:39:24.158303 5653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:39:24.159964 5653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:39:24.160025 5653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:39:24.160063 5653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:39:24.162227 5653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.240.84:38395/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:34325
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:39:24.162710 5653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:39:24.163022 5653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:39:24.169358 5659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:39:24.169682 5658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:39:24.169773 5661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:39:24.170998 5653 server_base.cc:1034] running on GCE node
I20260924 13:39:24.171197 5653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:39:24.171443 5653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:39:24.172628 5653 hybrid_clock.cc:648] HybridClock initialized: now 1790257164172594 us; error 46 us; skew 500 ppm
Sep 24 13:39:24 dist-test-slave-m5z1 krb5kdc[4822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257164, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:39:24.176138 5653 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:39:24.176659 5653 webserver.cc:460] Webserver started at http://127.25.240.67:44805/ using document root <none> and password file <none>
I20260924 13:39:24.176908 5653 fs_manager.cc:359] Metadata directory not provided
I20260924 13:39:24.176966 5653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:39:24.177186 5653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:39:24 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257164, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:39:24.196429 5653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "fd503efc3179412cbf6abc5413201356"
format_stamp: "Formatted at 2026-09-24 13:39:24 on dist-test-slave-m5z1"
server_key: "6c2dfe505c65cffc222249056140f2fe"
server_key_iv: "5130286e406b89ca400966ee3f46bd3d"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:24.196883 5653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "fd503efc3179412cbf6abc5413201356"
format_stamp: "Formatted at 2026-09-24 13:39:24 on dist-test-slave-m5z1"
server_key: "6c2dfe505c65cffc222249056140f2fe"
server_key_iv: "5130286e406b89ca400966ee3f46bd3d"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:24.198763 5653 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 13:39:24.227749 5668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:24.228027 5653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 13:39:24.228120 5653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "fd503efc3179412cbf6abc5413201356"
format_stamp: "Formatted at 2026-09-24 13:39:24 on dist-test-slave-m5z1"
server_key: "6c2dfe505c65cffc222249056140f2fe"
server_key_iv: "5130286e406b89ca400966ee3f46bd3d"
server_key_version: "kuduclusterkey@0"
I20260924 13:39:24.228232 5653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:39:24.254998 5653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:39:24.257733 5653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:39:24.257936 5653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:39:24.258237 5653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:39:24.258617 5653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:39:24.258661 5653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:24.258688 5653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:39:24.258708 5653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:39:24.265146 5653 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:46073
I20260924 13:39:24.267040 5781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:46073 every 8 connection(s)
I20260924 13:39:24.267155 5653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:39:24.268148 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 5653
I20260924 13:39:24.268270 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:39:24.268484 26561 external_mini_cluster.cc:1428] Setting key 4607d47a764fe5d60808632f4b6ad8d4
Sep 24 13:39:24 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257164, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:39:24.279575 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.268668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:41027 (local address 127.25.240.126:34325)
0924 13:39:24.268736 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:24.268739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.269813 (+ 1074us) server_negotiation.cc:407] Connection header received
0924 13:39:24.270399 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.270400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.270412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.270498 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:39:24.271182 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.271972 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.273339 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.273641 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.273683 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.276195 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:39:24.276197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:39:24.276197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:39:24.276205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:39:24.277964 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:24.278443 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:24.278444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:24.278445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:24.278514 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:24.278816 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:24.278817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:24.278817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:24.278962 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:39:24.279041 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.279383 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.279479 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 13:39:24.279729 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.267971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:34325 (local address 127.25.240.67:41027)
0924 13:39:24.269666 (+ 1695us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.269717 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 13:39:24.270303 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:39:24.270566 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:39:24.270570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:39:24.270799 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:39:24.271085 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.271090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.272069 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.272070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.273236 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.273239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.273772 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.273773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.273929 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.274498 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:39:24.274508 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:39:24.276083 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:39:24.278065 (+ 1982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:39:24.278073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:39:24.278076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:39:24.278364 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:39:24.278594 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:39:24.278595 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:39:24.278595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:39:24.278735 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:39:24.279044 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:39:24.279050 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:39:24.279216 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 13:39:24.279338 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 13:39:24.279494 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1649,"thread_start_us":82,"threads_started":1}
I20260924 13:39:24.280206 5782 heartbeater.cc:347] Connected to a master server at 127.25.240.126:34325
I20260924 13:39:24.280303 5782 heartbeater.cc:464] Registering TS with master...
I20260924 13:39:24.280483 5782 heartbeater.cc:511] Master 127.25.240.126:34325 requested a full tablet report, sending...
I20260924 13:39:24.280845 5297 ts_manager.cc:194] Registered new tserver with Master: fd503efc3179412cbf6abc5413201356 (127.25.240.67:46073)
I20260924 13:39:24.281216 5297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:41027
I20260924 13:39:24.291821 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:39:24.302793 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.293479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55746 (local address 127.25.240.126:34325)
0924 13:39:24.293519 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:24.293521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.293530 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:39:24.293589 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.293590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.293600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.293669 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:39:24.294367 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.295128 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.296119 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.296428 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.296505 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.299531 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:39:24.299533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:39:24.299533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:39:24.299542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:39:24.301287 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:24.301891 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:24.301892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:24.301892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:24.301957 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:39:24.302200 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:39:24.302201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:39:24.302201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:39:24.302344 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:39:24.302452 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.302615 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.302698 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:39:24.305466 5297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:39:24.307734 5297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 24 13:39:24 dist-test-slave-m5z1 krb5kdc[4822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257163, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:39:24.328841 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.321425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:43993 (local address 127.0.0.1:43940)
0924 13:39:24.322302 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.322310 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:39:24.322347 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:39:24.324332 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:39:24.324333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:39:24.324335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:39:24.324366 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.324367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.325889 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.325889 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.327042 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.327044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.328063 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.328064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.328144 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.328159 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 13:39:24.328230 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:39:24.328282 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":63,"threads_started":1}
I20260924 13:39:24.329085 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.321641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43940 (local address 127.25.240.65:43993)
0924 13:39:24.321842 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:24.321845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.324072 (+ 2227us) server_negotiation.cc:407] Connection header received
0924 13:39:24.324111 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.324113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.324129 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.324209 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:39:24.324457 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.325464 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.327144 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.327881 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.327966 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.328026 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.328975 (+ 949us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.329017 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":58,"threads_started":1}
I20260924 13:39:24.330075 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 9973eced271443d48a1a0dc48c6f8ef7 (DEFAULT_TABLE table=test-table [id=c6c6c50b58594857976dd7bf65c41273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:39:24.330431 5436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9973eced271443d48a1a0dc48c6f8ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:39:24.333215 5801 tablet_bootstrap.cc:492] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb: Bootstrap starting.
I20260924 13:39:24.334613 5801 tablet_bootstrap.cc:654] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb: Neither blocks nor log segments found. Creating new log.
I20260924 13:39:24.334986 5801 log.cc:824] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb: Log is configured to *not* fsync() on all Append() calls
I20260924 13:39:24.338526 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.321389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37288 (local address 127.25.240.66:44739)
0924 13:39:24.321971 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:24.321976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.322104 (+ 128us) server_negotiation.cc:407] Connection header received
0924 13:39:24.322141 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.322143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.322157 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.322215 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:39:24.322500 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.323785 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.337170 (+ 13385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.338046 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.338097 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.338164 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.338401 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.338442 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":72,"threads_started":1}
I20260924 13:39:24.338719 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.321279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:44739 (local address 127.0.0.1:37288)
0924 13:39:24.322013 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.322021 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:39:24.322061 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:39:24.322395 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:39:24.322395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:39:24.322397 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:39:24.322437 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.322439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.323859 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.323860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.337030 (+ 13170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.337033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.338229 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.338230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.338296 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.338301 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:39:24.338608 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 13:39:24.338653 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":11783,"thread_start_us":63,"threads_started":1}
I20260924 13:39:24.339423 5582 tablet_service.cc:1467] Processing CreateTablet for tablet 9973eced271443d48a1a0dc48c6f8ef7 (DEFAULT_TABLE table=test-table [id=c6c6c50b58594857976dd7bf65c41273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:39:24.339785 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9973eced271443d48a1a0dc48c6f8ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:39:24.344375 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.321049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:46073 (local address 127.0.0.1:39282)
0924 13:39:24.321714 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.321756 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:39:24.321826 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:39:24.322160 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:39:24.322163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:39:24.322166 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:39:24.322245 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.322247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.323931 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.323932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.335972 (+ 12040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.335976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.344096 (+ 8120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.344097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.344176 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.344181 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:39:24.344245 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:39:24.344293 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":223,"thread_start_us":75,"threads_started":1}
I20260924 13:39:24.346390 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.321151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.25.240.67:46073)
0924 13:39:24.321333 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:24.321759 (+ 426us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.321807 (+ 48us) server_negotiation.cc:407] Connection header received
0924 13:39:24.321920 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.321923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.321941 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.322064 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:39:24.322332 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.323825 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.337200 (+ 13375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.338009 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.338090 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.338163 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.346247 (+ 8084us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.346307 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":67,"threads_started":1}
I20260924 13:39:24.346760 5803 tablet_bootstrap.cc:492] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: Bootstrap starting.
I20260924 13:39:24.347196 5716 tablet_service.cc:1467] Processing CreateTablet for tablet 9973eced271443d48a1a0dc48c6f8ef7 (DEFAULT_TABLE table=test-table [id=c6c6c50b58594857976dd7bf65c41273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:39:24.347574 5716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9973eced271443d48a1a0dc48c6f8ef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:39:24.347918 5803 tablet_bootstrap.cc:654] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: Neither blocks nor log segments found. Creating new log.
I20260924 13:39:24.348263 5803 log.cc:824] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: Log is configured to *not* fsync() on all Append() calls
I20260924 13:39:24.349208 5801 tablet_bootstrap.cc:492] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb: No bootstrap required, opened a new log
I20260924 13:39:24.349336 5801 ts_tablet_manager.cc:1397] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260924 13:39:24.351815 5805 tablet_bootstrap.cc:492] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356: Bootstrap starting.
I20260924 13:39:24.352860 5805 tablet_bootstrap.cc:654] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356: Neither blocks nor log segments found. Creating new log.
I20260924 13:39:24.353670 5805 log.cc:824] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356: Log is configured to *not* fsync() on all Append() calls
I20260924 13:39:24.354434 5805 tablet_bootstrap.cc:492] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356: No bootstrap required, opened a new log
I20260924 13:39:24.354522 5805 ts_tablet_manager.cc:1397] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 13:39:24.354811 5801 raft_consensus.cc:348] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.356412 5805 raft_consensus.cc:348] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.356590 5805 raft_consensus.cc:374] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:39:24.356631 5805 raft_consensus.cc:729] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd503efc3179412cbf6abc5413201356, State: Initialized, Role: FOLLOWER
I20260924 13:39:24.356724 5801 raft_consensus.cc:374] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:39:24.356747 5805 consensus_queue.cc:260] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.356882 5801 raft_consensus.cc:729] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6521e005ac084fc4a2e58a7207a425fb, State: Initialized, Role: FOLLOWER
I20260924 13:39:24.357010 5805 ts_tablet_manager.cc:1428] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 13:39:24.357046 5801 consensus_queue.cc:260] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.357194 5782 heartbeater.cc:503] Master 127.25.240.126:34325 was elected leader, sending a full tablet report...
I20260924 13:39:24.357400 5801 ts_tablet_manager.cc:1428] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260924 13:39:24.357577 5512 heartbeater.cc:503] Master 127.25.240.126:34325 was elected leader, sending a full tablet report...
I20260924 13:39:24.359355 5803 tablet_bootstrap.cc:492] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: No bootstrap required, opened a new log
I20260924 13:39:24.359457 5803 ts_tablet_manager.cc:1397] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260924 13:39:24.360793 5803 raft_consensus.cc:348] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.360939 5803 raft_consensus.cc:374] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:39:24.360991 5803 raft_consensus.cc:729] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869564eb1cdf418db751cd2feb7342f5, State: Initialized, Role: FOLLOWER
I20260924 13:39:24.361107 5803 consensus_queue.cc:260] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.361388 5648 heartbeater.cc:503] Master 127.25.240.126:34325 was elected leader, sending a full tablet report...
I20260924 13:39:24.361626 5803 ts_tablet_manager.cc:1428] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:39:24.395442 5807 raft_consensus.cc:482] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:39:24.395593 5807 raft_consensus.cc:504] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.396018 5807 leader_election.cc:288] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869564eb1cdf418db751cd2feb7342f5 (127.25.240.66:44739), 6521e005ac084fc4a2e58a7207a425fb (127.25.240.65:43993)
I20260924 13:39:24.400341 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.396241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:44739 (local address 127.25.240.67:43481)
0924 13:39:24.396517 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.396529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:39:24.396571 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:39:24.397005 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:39:24.397006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:39:24.397008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:39:24.397081 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.397082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.398082 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.398083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.399278 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.399281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.400092 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.400093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.400172 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.400176 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:39:24.400228 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 13:39:24.400266 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260924 13:39:24.400843 5798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.396706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:43481 (local address 127.25.240.66:44739)
0924 13:39:24.396748 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:24.396749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.396757 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:39:24.396781 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.396782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.396793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.396837 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:39:24.397172 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.397948 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.399427 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.399972 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.400047 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.400058 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.400738 (+ 680us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.400780 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:39:24.401149 5602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9973eced271443d48a1a0dc48c6f8ef7" candidate_uuid: "fd503efc3179412cbf6abc5413201356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869564eb1cdf418db751cd2feb7342f5" is_pre_election: true
I20260924 13:39:24.401325 5602 raft_consensus.cc:2454] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd503efc3179412cbf6abc5413201356 in term 0.
I20260924 13:39:24.401597 5669 leader_election.cc:302] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869564eb1cdf418db751cd2feb7342f5, fd503efc3179412cbf6abc5413201356; no voters:
I20260924 13:39:24.401743 5807 raft_consensus.cc:2781] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:39:24.401796 5807 raft_consensus.cc:482] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:39:24.401827 5807 raft_consensus.cc:3037] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:39:24.402717 5807 raft_consensus.cc:504] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.402865 5807 leader_election.cc:288] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [CANDIDATE]: Term 1 election: Requested vote from peers 869564eb1cdf418db751cd2feb7342f5 (127.25.240.66:44739), 6521e005ac084fc4a2e58a7207a425fb (127.25.240.65:43993)
I20260924 13:39:24.403076 5602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9973eced271443d48a1a0dc48c6f8ef7" candidate_uuid: "fd503efc3179412cbf6abc5413201356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869564eb1cdf418db751cd2feb7342f5"
I20260924 13:39:24.403167 5602 raft_consensus.cc:3037] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:39:24.404042 5602 raft_consensus.cc:2454] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd503efc3179412cbf6abc5413201356 in term 1.
I20260924 13:39:24.404234 5669 leader_election.cc:302] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869564eb1cdf418db751cd2feb7342f5, fd503efc3179412cbf6abc5413201356; no voters:
I20260924 13:39:24.404376 5807 raft_consensus.cc:2781] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:39:24.404529 5807 raft_consensus.cc:686] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 1 LEADER]: Becoming Leader. State: Replica: fd503efc3179412cbf6abc5413201356, State: Running, Role: LEADER
I20260924 13:39:24.404614 5807 consensus_queue.cc:237] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } }
I20260924 13:39:24.405467 5297 catalog_manager.cc:5495] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd503efc3179412cbf6abc5413201356 (127.25.240.67). New cstate: current_term: 1 leader_uuid: "fd503efc3179412cbf6abc5413201356" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd503efc3179412cbf6abc5413201356" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 46073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 } health_report { overall_health: UNKNOWN } } }
I20260924 13:39:24.407924 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.396325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:43993 (local address 127.25.240.67:51267)
0924 13:39:24.400421 (+ 4096us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.400433 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:39:24.400481 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:39:24.404243 (+ 3762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:39:24.404244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:39:24.404247 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:39:24.404281 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.404282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.405164 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.405165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.406565 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:39:24.406568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.407708 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.407709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:39:24.407770 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.407773 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:39:24.407823 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:39:24.407856 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4085,"thread_start_us":96,"threads_started":1}
I20260924 13:39:24.408013 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.396919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:51267 (local address 127.25.240.65:43993)
0924 13:39:24.396958 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:24.396960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.400535 (+ 3575us) server_negotiation.cc:407] Connection header received
0924 13:39:24.400564 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.400565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.400577 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.400618 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:39:24.404368 (+ 3750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.405070 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.406649 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.407225 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.407310 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.407320 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.407888 (+ 568us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.407940 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:39:24.408291 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9973eced271443d48a1a0dc48c6f8ef7" candidate_uuid: "fd503efc3179412cbf6abc5413201356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521e005ac084fc4a2e58a7207a425fb" is_pre_election: true
I20260924 13:39:24.408445 5464 raft_consensus.cc:2454] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd503efc3179412cbf6abc5413201356 in term 0.
I20260924 13:39:24.408560 5465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9973eced271443d48a1a0dc48c6f8ef7" candidate_uuid: "fd503efc3179412cbf6abc5413201356" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521e005ac084fc4a2e58a7207a425fb"
I20260924 13:39:24.408617 5465 raft_consensus.cc:3037] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:39:24.409449 5465 raft_consensus.cc:2454] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd503efc3179412cbf6abc5413201356 in term 1.
I20260924 13:39:24.418908 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.416002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.25.240.67:46073)
0924 13:39:24.416045 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:39:24.416047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:39:24.416056 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:39:24.416090 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:39:24.416091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:39:24.416113 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:39:24.416186 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:39:24.416303 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.416975 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.418012 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:39:24.418295 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:39:24.418365 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:39:24.418539 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:39:24.418662 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:39:24.418745 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:39:24.418790 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 13:39:24.418812 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:39:24.420856 5465 raft_consensus.cc:1261] T 9973eced271443d48a1a0dc48c6f8ef7 P 6521e005ac084fc4a2e58a7207a425fb [term 1 FOLLOWER]: Refusing update from remote peer fd503efc3179412cbf6abc5413201356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:39:24.420856 5602 raft_consensus.cc:1261] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 1 FOLLOWER]: Refusing update from remote peer fd503efc3179412cbf6abc5413201356: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:39:24.421236 5811 consensus_queue.cc:1035] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6521e005ac084fc4a2e58a7207a425fb" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 43993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:39:24.421442 5807 consensus_queue.cc:1035] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869564eb1cdf418db751cd2feb7342f5" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:39:24.424031 5814 mvcc.cc:205] Tried to move back new op lower bound from 7332893345465311232 to 7332893345401798656. Current Snapshot: MvccSnapshot[applied={T|T < 7332893345465311232}]
I20260924 13:39:24.426061 5815 mvcc.cc:205] Tried to move back new op lower bound from 7332893345465311232 to 7332893345401798656. Current Snapshot: MvccSnapshot[applied={T|T < 7332893345465311232}]
I20260924 13:39:24.426399 5296 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:39:24.426486 5296 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55746:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:39:24.427290 5296 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f99420dff814186b65c0c1e5b9d7469: Sending DeleteTablet for 3 replicas of tablet 9973eced271443d48a1a0dc48c6f8ef7
I20260924 13:39:24.427721 5716 tablet_service.cc:1514] Processing DeleteTablet for tablet 9973eced271443d48a1a0dc48c6f8ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:39:24 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:39282
I20260924 13:39:24.427847 5817 mvcc.cc:205] Tried to move back new op lower bound from 7332893345465311232 to 7332893345401798656. Current Snapshot: MvccSnapshot[applied={T|T < 7332893345465311232}]
I20260924 13:39:24.427943 5582 tablet_service.cc:1514] Processing DeleteTablet for tablet 9973eced271443d48a1a0dc48c6f8ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:39:24 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:37288
I20260924 13:39:24.428170 5825 tablet_replica.cc:330] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: stopping tablet replica
I20260924 13:39:24.428259 5825 raft_consensus.cc:2229] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:39:24.428288 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 9973eced271443d48a1a0dc48c6f8ef7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:39:24 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:43940
I20260924 13:39:24.428354 5825 raft_consensus.cc:2258] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:39:24.428977 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 5379
I20260924 13:39:24.432163 5825 ts_tablet_manager.cc:1905] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:39:24.433962 5825 ts_tablet_manager.cc:1918] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:39:24.434104 5825 log.cc:1196] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/9973eced271443d48a1a0dc48c6f8ef7
I20260924 13:39:24.434775 5825 ts_tablet_manager.cc:1939] T 9973eced271443d48a1a0dc48c6f8ef7 P 869564eb1cdf418db751cd2feb7342f5: Deleting consensus metadata
I20260924 13:39:24.435299 5282 catalog_manager.cc:4841] TS 869564eb1cdf418db751cd2feb7342f5 (127.25.240.66:44739): tablet 9973eced271443d48a1a0dc48c6f8ef7 (table test-table [id=c6c6c50b58594857976dd7bf65c41273]) successfully deleted
W20260924 13:39:24.437604 5282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:43993 (error 108)
I20260924 13:39:24.438115 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.437974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:43993 (local address 127.0.0.1:43948)
0924 13:39:24.438041 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.438081 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:43993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 13:39:24.438225 5282 catalog_manager.cc:4584] TS 6521e005ac084fc4a2e58a7207a425fb (127.25.240.65:43993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9973eced271443d48a1a0dc48c6f8ef7: Network error: Client connection negotiation failed: client connection to 127.25.240.65:43993: connect: Connection refused (error 111)
I20260924 13:39:24.438459 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 5518
I20260924 13:39:24.440503 5824 tablet_replica.cc:330] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356: stopping tablet replica
I20260924 13:39:24.440625 5824 raft_consensus.cc:2229] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:39:24.440723 5824 raft_consensus.cc:2258] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:39:24.441149 5824 ts_tablet_manager.cc:1905] T 9973eced271443d48a1a0dc48c6f8ef7 P fd503efc3179412cbf6abc5413201356: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:39:24.442655 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 5653
I20260924 13:39:24.446985 5795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:39:24.446870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:46073 (local address 127.0.0.1:39308)
0924 13:39:24.446922 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 13:39:24.446947 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.67:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 13:39:24.447417 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 5266
I20260924 13:39:24.452549 26561 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 13:39:24.797003 26561 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 13:39:24.797106 26561 mini_ranger.cc:66] Stopping Ranger...
I20260924 13:39:25.132833 26561 mini_ranger.cc:68] Stopped Ranger
2026-09-24 13:39:25.133 UTC [4852] LOG: received smart shutdown request
2026-09-24 13:39:25.136 UTC [4852] LOG: background worker "logical replication launcher" (PID 4861) exited with exit code 1
2026-09-24 13:39:25.136 UTC [4856] LOG: shutting down
2026-09-24 13:39:25.307 UTC [4852] LOG: database system is shut down
2026-09-24T13:39:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44995 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:39:25 dist-test-slave-m5z1 krb5kdc[5836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:39:25 dist-test-slave-m5z1 krb5kdc[5836](info): set up 2 sockets
Sep 24 13:39:25 dist-test-slave-m5z1 krb5kdc[5836](info): commencing operation
krb5kdc: starting...
W20260924 13:39:26.457849 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:39:26 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:39:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:39:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/rangeradmin_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/rangeradmin_127.25.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/rangerlookup_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/rangerlookup_127.25.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/HTTP_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/HTTP_127.25.240.84@KRBTEST.COM.keytab.
I20260924 13:39:26.623713 26561 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFHYdCJ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/postgres -l logfile start
2026-09-24 13:39:29.188 UTC [5866] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 13:39:29.189 UTC [5866] LOG: listening on IPv4 address "127.25.240.84", port 40693
2026-09-24 13:39:29.193 UTC [5866] LOG: listening on Unix socket "/tmp/.s.PGSQL.40693"
2026-09-24 13:39:29.221 UTC [5869] LOG: database system was shut down at 2026-09-24 13:39:27 UTC
2026-09-24 13:39:29.224 UTC [5866] LOG: database system is ready to accept connections
I20260924 13:39:30.185654 26561 mini_postgres.cc:96] Postgres bound to 40693
2026-09-24 13:39:30.191 UTC [5877] FATAL: database "slave" does not exist
127.25.240.84:40693 - accepting connections
I20260924 13:39:30.192220 26561 mini_ranger.cc:161] Starting Ranger...
I20260924 13:39:30.206761 26561 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 13:39:30.679322 26561 mini_ranger.cc:87] Created ranger Postgres database
I20260924 13:39:30.679407 26561 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-admin
2026-09-24 13:39:30,967 [I] DB FLAVOR :POSTGRES
2026-09-24 13:39:30,967 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:39:30,967 [I] Checking connection..
2026-09-24 13:39:30,967 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 13:39:31,219 [I] Checking connection passed.
2026-09-24 13:39:31,219 [I] --------- Verifying version history table ---------
2026-09-24 13:39:31,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:39:31,456 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:39:31,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:39:31,713 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 13:39:31,713 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 13:39:31,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 13:39:31.937 UTC [5941] WARNING: there is no transaction in progress
2026-09-24 13:39:31,943 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 13:39:31,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 13:39:32,171 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 13:39:32,171 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:39:32,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 13:39:32,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 13:39:32,674 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m5z1','N') ;"
2026-09-24 13:39:32,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:39:33,173 [I] Table x_portal_user does not exist in database ranger
2026-09-24 13:39:33,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:39:33,434 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 13:39:33,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:39:33,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:39:33,909 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 13:39:33,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 13:39:34.892 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:34.908 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:34.925 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:34.940 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:34.956 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:34.976 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.038 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.045 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.059 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.069 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.079 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.093 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.102 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.108 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.114 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35.121 UTC [6049] WARNING: there is no transaction in progress
2026-09-24 13:39:35,779 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 13:39:35,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 13:39:36,009 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 13:39:36,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 13:39:36,255 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 13:39:36,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 13:39:36,514 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 13:39:36,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 13:39:36,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 13:39:37,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m5z1';"
2026-09-24 13:39:37,231 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 13:39:37,232 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 13:39:37,482 [I] DB_PATCHES have already been applied
I20260924 13:39:37.487933 26561 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFHYdCJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 13:39:37.487998 26561 mini_ranger.cc:193] Using host: 127.25.240.84
I20260924 13:39:37.491134 26561 mini_ranger.cc:227] Ranger admin URL: http://127.25.240.84:45881
Sep 24, 2026 1:39:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:39:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:39:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 1:39:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 1:39:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 1:39:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 1:39:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 13:39:38.622553 26561 mini_ranger.cc:160] Time spent starting Ranger: real 8.430s user 0.000s sys 0.008s
Sep 24, 2026 1:39:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 1:39:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 1:39:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 1:39:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 1:39:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 13:39:59.241550 26561 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.240.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.240.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/rangerkms_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.240.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/rangerkms_127.25.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.240.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.240.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.240.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/HTTP_127.25.240.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.240.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/HTTP_127.25.240.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 13:39:59 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257199, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 13:39:59.323881 26561 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 13:39:59.333127 26561 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 13:40:00.250187 26561 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 13:40:00.250274 26561 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms
I20260924 13:40:00.250294 26561 mini_ranger_kms.cc:227] Using postgres at 127.25.240.84:40693
2026-09-24 13:40:00,415 [I] DB FLAVOR :POSTGRES
2026-09-24 13:40:00,415 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 13:40:00,415 [I] Checking connection
2026-09-24 13:40:00,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 13:40:00,672 [I] connection success
2026-09-24 13:40:00,672 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 13:40:00,672 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 13:40:00,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 13:40:00,931 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 13:40:00,932 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 13:40:00,932 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 13:40:00,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.240.84:40693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 13:40:01,195 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 13:40:01.654670 26561 mini_ranger_kms.cc:310] Created kms service
I20260924 13:40:01.878691 26561 mini_ranger_kms.cc:326] Created kudu user
I20260924 13:40:01.925941 26561 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 13:40:01 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257199, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:02.354270 26561 mini_ranger_kms.cc:384] Added ranger policy
I20260924 13:40:02.354393 26561 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFHYdCJ/build/release/bin/postgresql.jar:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFHYdCJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 13:40:02.354413 26561 mini_ranger_kms.cc:240] Using host: 127.25.240.84
I20260924 13:40:02.357872 26561 mini_ranger_kms.cc:276] Ranger KMS PID: 6229
I20260924 13:40:02.357950 26561 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.240.84:33903
Sep 24, 2026 1:40:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 1:40:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 1:40:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 1:40:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 1:40:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 1:40:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 13:40:04.613677 26561 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.290s user 0.002s sys 0.147s
I20260924 13:40:04.613771 26561 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 13:40:04.613816 26561 mini_ranger_kms.cc:401] 127.25.240.84:33903/kms/v1/keys
Sep 24 13:40:04 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257204, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.240.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 13:40:05 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257204, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.240.84@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
Sep 24 13:40:05 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257204, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.240.84@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
Sep 24 13:40:08 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:40:08.066749 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:42795
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:42795
--ranger_config_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:33903/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:08.147056 6280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:08.147255 6280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:08.147277 6280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:08.148371 6280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:08.148407 6280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:08.148422 6280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:08.148433 6280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:08.148447 6280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:08.149928 6280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.240.84:33903/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:42795
--ranger_config_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:08.150117 6280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:08.150280 6280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:08.155493 6285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:08.155519 6288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:08.155525 6280 server_base.cc:1034] running on GCE node
W20260924 13:40:08.155486 6286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:08.155927 6280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:08.156180 6280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:08.157312 6280 hybrid_clock.cc:648] HybridClock initialized: now 1790257208157299 us; error 29 us; skew 500 ppm
Sep 24 13:40:08 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257208, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:08.159772 6280 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:08.160212 6280 webserver.cc:460] Webserver started at http://127.25.240.126:41471/ using document root <none> and password file <none>
I20260924 13:40:08.160444 6280 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:08.160483 6280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:08.160631 6280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:40:08 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257208, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:08.219033 6280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "9f090c2ea20c482e87db7a121244e77a"
format_stamp: "Formatted at 2026-09-24 13:40:08 on dist-test-slave-m5z1"
server_key: "68e6c9adf012181ca2b485983af337fb"
server_key_iv: "9b9b1af7e758de12e6729e319a826802"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:08.219416 6280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "9f090c2ea20c482e87db7a121244e77a"
format_stamp: "Formatted at 2026-09-24 13:40:08 on dist-test-slave-m5z1"
server_key: "68e6c9adf012181ca2b485983af337fb"
server_key_iv: "9b9b1af7e758de12e6729e319a826802"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:08.220888 6280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 13:40:08.232838 6295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:08.233049 6280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:08.233151 6280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "9f090c2ea20c482e87db7a121244e77a"
format_stamp: "Formatted at 2026-09-24 13:40:08 on dist-test-slave-m5z1"
server_key: "68e6c9adf012181ca2b485983af337fb"
server_key_iv: "9b9b1af7e758de12e6729e319a826802"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:08.233253 6280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:08.254038 6280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:08.256106 6280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:08.256266 6280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:08.260747 6280 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:42795
I20260924 13:40:08.260784 6347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:42795 every 8 connection(s)
I20260924 13:40:08.261183 6280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:08.261986 6348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:08.262303 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 6280
I20260924 13:40:08.262406 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:40:08.262615 26561 external_mini_cluster.cc:1428] Setting key 42cce387da383236889eafb210d91dd1
I20260924 13:40:08.265043 6348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Bootstrap starting.
I20260924 13:40:08.265791 6348 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:08.266111 6348 log.cc:824] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:08.266767 6348 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: No bootstrap required, opened a new log
I20260924 13:40:08.268520 6348 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } }
I20260924 13:40:08.268669 6348 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:08.268714 6348 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f090c2ea20c482e87db7a121244e77a, State: Initialized, Role: FOLLOWER
I20260924 13:40:08.268841 6348 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } }
I20260924 13:40:08.268930 6348 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:40:08.268980 6348 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 24 13:40:08 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:08.269029 6348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:08.269793 6348 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } }
I20260924 13:40:08.269923 6348 leader_election.cc:302] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f090c2ea20c482e87db7a121244e77a; no voters:
I20260924 13:40:08.270115 6348 leader_election.cc:288] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:40:08.270252 6353 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:08.270366 6353 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 1 LEADER]: Becoming Leader. State: Replica: 9f090c2ea20c482e87db7a121244e77a, State: Running, Role: LEADER
I20260924 13:40:08.270372 6348 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:08.271113 6353 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } }
I20260924 13:40:08.271512 6355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f090c2ea20c482e87db7a121244e77a. Latest consensus state: current_term: 1 leader_uuid: "9f090c2ea20c482e87db7a121244e77a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } } }
I20260924 13:40:08.271607 6355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:08.271776 6354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f090c2ea20c482e87db7a121244e77a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } } }
I20260924 13:40:08.271848 6354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:08.271912 6348 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 13:40:08.271924 6367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:08.272408 6367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:08.276171 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:08.263130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.25.240.126:42795)
0924 13:40:08.263303 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:08.263313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:08.263386 (+ 73us) server_negotiation.cc:407] Connection header received
0924 13:40:08.263880 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:08.263887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:08.263934 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:08.264092 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:08.264696 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:08.265707 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:08.266551 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:08.267004 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:08.267134 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:08.269522 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:08.269525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:08.269526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:08.269537 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:08.271428 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:08.271878 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:08.271881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:08.271881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:08.271980 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:08.272219 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:08.272219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:08.272220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:08.272518 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:08.275126 (+ 2608us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:08.275339 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 13:40:08.275826 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260924 13:40:08.278337 6367 catalog_manager.cc:1347] Generated new cluster ID: 481adfa8a5f64f88ad1ffdcef435fbc7
I20260924 13:40:08.278402 6367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:40:08.288869 6367 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:40:08.289705 6367 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:40:08.303380 6367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Generated new TSK 0
I20260924 13:40:08.303571 6367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:40:09.002633 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:42795
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:09.106076 6393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:09.106273 6393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:09.106297 6393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:09.107466 6393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:09.107509 6393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:09.107535 6393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:40:09.109073 6393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.240.84:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:42795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:09.109302 6393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:09.109548 6393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:09.115113 6398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:09.115238 6401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:09.115123 6399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:09.116466 6393 server_base.cc:1034] running on GCE node
I20260924 13:40:09.116686 6393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:09.116947 6393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:09.118156 6393 hybrid_clock.cc:648] HybridClock initialized: now 1790257209118130 us; error 39 us; skew 500 ppm
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:09.121317 6393 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:40:09.121881 6393 webserver.cc:460] Webserver started at http://127.25.240.65:45633/ using document root <none> and password file <none>
I20260924 13:40:09.122164 6393 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:09.122237 6393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:09.122381 6393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:09.137174 6393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "5f3e33afc05c4a208d39b668ec976490"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "aba4dd66d379b3363fcf7ff2df5520e0"
server_key_iv: "8b9a9533900fb8473f5637968ee6e357"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.137782 6393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "5f3e33afc05c4a208d39b668ec976490"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "aba4dd66d379b3363fcf7ff2df5520e0"
server_key_iv: "8b9a9533900fb8473f5637968ee6e357"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.142041 6393 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.002s
I20260924 13:40:09.157706 6410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.157963 6393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260924 13:40:09.158072 6393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "5f3e33afc05c4a208d39b668ec976490"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "aba4dd66d379b3363fcf7ff2df5520e0"
server_key_iv: "8b9a9533900fb8473f5637968ee6e357"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.158174 6393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:09.192420 6393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:09.195016 6393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:09.195214 6393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:09.195477 6393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:09.195880 6393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:09.195969 6393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.196065 6393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:09.196137 6393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.202310 6393 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:34009
I20260924 13:40:09.202806 6393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:40:09.206034 6523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:34009 every 8 connection(s)
I20260924 13:40:09.209437 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 6393
I20260924 13:40:09.209551 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:40:09.209801 26561 external_mini_cluster.cc:1428] Setting key 818ef74cf953991c15e555d8f57f0aca
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:09.217660 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:09.206707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:44843 (local address 127.25.240.126:42795)
0924 13:40:09.208002 (+ 1295us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:09.208004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:09.208012 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:40:09.208038 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:09.208039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:09.208051 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:09.208114 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:09.208869 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.209711 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.211182 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.211528 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.211607 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:09.214230 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:09.214231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:09.214232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:09.214239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:09.216100 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:09.216592 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:09.216593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:09.216593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:09.216662 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:09.216932 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:09.216933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:09.216933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:09.217071 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:09.217119 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:09.217442 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 13:40:09.217533 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1266,"thread_start_us":103,"threads_started":1}
I20260924 13:40:09.217979 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:09.206690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:42795 (local address 127.25.240.65:44843)
0924 13:40:09.207127 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:09.207174 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:09.207815 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:09.208239 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:09.208244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:09.208477 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:09.208760 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:09.208765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.209814 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.209815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:09.211083 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:09.211086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.211646 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.211648 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:09.211791 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:09.212386 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:09.212396 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:09.214113 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:09.216196 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:09.216203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:09.216205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:09.216491 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:09.216732 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:09.216732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:09.216733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:09.216857 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:09.217176 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:09.217181 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:09.217300 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 13:40:09.217418 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 13:40:09.217593 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":97,"threads_started":1}
I20260924 13:40:09.218477 6524 heartbeater.cc:347] Connected to a master server at 127.25.240.126:42795
I20260924 13:40:09.218571 6524 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:09.218750 6524 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:09.219360 6312 ts_manager.cc:194] Registered new tserver with Master: 5f3e33afc05c4a208d39b668ec976490 (127.25.240.65:34009)
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
I20260924 13:40:09.220255 6312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:44843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:40:09.241210 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:42795
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 13:40:09.359625 6532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:09.359899 6532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:09.359936 6532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:09.361693 6532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:09.361825 6532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:09.361917 6532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:40:09.364148 6532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.240.84:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:42795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:09.364410 6532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:09.364733 6532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:09.371973 6540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:09.372078 6537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:09.371989 6538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:09.372335 6532 server_base.cc:1034] running on GCE node
I20260924 13:40:09.372525 6532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:09.372774 6532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:09.373930 6532 hybrid_clock.cc:648] HybridClock initialized: now 1790257209373916 us; error 31 us; skew 500 ppm
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:09.376523 6532 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:40:09.377007 6532 webserver.cc:460] Webserver started at http://127.25.240.66:35249/ using document root <none> and password file <none>
I20260924 13:40:09.377262 6532 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:09.377327 6532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:09.377463 6532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:09.392937 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "7243f028684b4c1d9cd7f24cd56db6e0"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "c58aac20966d3f7726197400eb577412"
server_key_iv: "15c18ae38df971c3bdd49520df0dc168"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.393344 6532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "7243f028684b4c1d9cd7f24cd56db6e0"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "c58aac20966d3f7726197400eb577412"
server_key_iv: "15c18ae38df971c3bdd49520df0dc168"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.395334 6532 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 13:40:09.414628 6548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.414894 6532 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:09.415022 6532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "7243f028684b4c1d9cd7f24cd56db6e0"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "c58aac20966d3f7726197400eb577412"
server_key_iv: "15c18ae38df971c3bdd49520df0dc168"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.415128 6532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:09.437229 6532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:09.439405 6532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:09.439587 6532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:09.439837 6532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:09.440183 6532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:09.440246 6532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.440315 6532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:09.440344 6532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.447012 6532 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:45305
I20260924 13:40:09.447515 6532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:40:09.449128 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 6532
I20260924 13:40:09.449246 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:40:09.449479 26561 external_mini_cluster.cc:1428] Setting key efa0860abc47155d0c335e2ac17d5e38
I20260924 13:40:09.449975 6661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:45305 every 8 connection(s)
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:09.467393 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:09.448872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:58333 (local address 127.25.240.126:42795)
0924 13:40:09.448927 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:09.448930 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:09.457442 (+ 8512us) server_negotiation.cc:407] Connection header received
0924 13:40:09.458203 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:09.458204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:09.458219 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:09.458251 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:09.459135 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.459911 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.461301 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.461680 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.461728 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:09.464333 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:09.464335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:09.464335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:09.464343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:09.465806 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:09.466294 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:09.466295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:09.466296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:09.466358 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:09.466658 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:09.466659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:09.466659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:09.466808 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:09.466875 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:09.467197 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 13:40:09.467292 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:40:09.467885 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:09.448863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:42795 (local address 127.25.240.66:58333)
0924 13:40:09.457294 (+ 8431us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:09.457346 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:09.458070 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:09.458468 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:09.458473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:09.458726 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:09.459014 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:09.459022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.460034 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.460034 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:09.461206 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:09.461209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.461829 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.461830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:09.461971 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:09.462556 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:09.462567 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:09.464240 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:09.465898 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:09.465904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:09.465907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:09.466199 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:09.466431 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:09.466432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:09.466432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:09.466572 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:09.466865 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:09.466870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:09.467030 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 13:40:09.467490 (+ 460us) client_negotiation.cc:240] Negotiation successful
0924 13:40:09.467657 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8375,"thread_start_us":606,"threads_started":1}
I20260924 13:40:09.468328 6662 heartbeater.cc:347] Connected to a master server at 127.25.240.126:42795
I20260924 13:40:09.468415 6662 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:09.468602 6662 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:09.468933 6312 ts_manager.cc:194] Registered new tserver with Master: 7243f028684b4c1d9cd7f24cd56db6e0 (127.25.240.66:45305)
I20260924 13:40:09.469405 6312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:58333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:40:09.495133 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:42795
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:09.621944 6667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:09.622186 6667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:09.622225 6667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:09.623682 6667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:09.623749 6667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:09.623795 6667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:40:09.625309 6667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.240.84:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:42795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:09.625566 6667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:09.625857 6667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:09.630892 6675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:09.631146 6672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:09.631316 6673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:09.632236 6667 server_base.cc:1034] running on GCE node
I20260924 13:40:09.632442 6667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:09.632714 6667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:09.633864 6667 hybrid_clock.cc:648] HybridClock initialized: now 1790257209633845 us; error 32 us; skew 500 ppm
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:09.637180 6667 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:40:09.637712 6667 webserver.cc:460] Webserver started at http://127.25.240.67:45785/ using document root <none> and password file <none>
I20260924 13:40:09.637962 6667 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:09.638018 6667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:09.638146 6667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:09.655573 6667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "eb078d61dfd443e693e67d3c1f1ece2c"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "57934863edc2e6a349b4d86537aa8f3f"
server_key_iv: "465505b6eebd23daed4cf40084c1cd79"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.656046 6667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "eb078d61dfd443e693e67d3c1f1ece2c"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "57934863edc2e6a349b4d86537aa8f3f"
server_key_iv: "465505b6eebd23daed4cf40084c1cd79"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.657495 6667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:40:09.681751 6682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.682039 6667 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.001s sys 0.000s
I20260924 13:40:09.682134 6667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "eb078d61dfd443e693e67d3c1f1ece2c"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "57934863edc2e6a349b4d86537aa8f3f"
server_key_iv: "465505b6eebd23daed4cf40084c1cd79"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.682228 6667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:09.711126 6667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:09.713418 6667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:09.713621 6667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:09.713888 6667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:09.714246 6667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:09.714294 6667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.714375 6667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:09.714406 6667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.721928 6667 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:44641
I20260924 13:40:09.722355 6667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:40:09.726632 6795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:44641 every 8 connection(s)
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:09.730209 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 6667
I20260924 13:40:09.730329 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:40:09.730540 26561 external_mini_cluster.cc:1428] Setting key 7db96249c7e8cc89639ef24f1d80a515
I20260924 13:40:09.732182 6527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:09.723781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:44511 (local address 127.25.240.126:42795)
0924 13:40:09.723833 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:09.723835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:09.723844 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:40:09.724330 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:09.724332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:09.724345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:09.724392 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:09.724942 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.725451 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.726500 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.726730 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.726793 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:09.729029 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:09.729030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:09.729030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:09.729039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:09.730680 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:09.731179 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:09.731180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:09.731180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:09.731237 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:09.731507 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:09.731508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:09.731508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:09.731625 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:09.731698 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:09.732009 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 13:40:09.732092 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:40:09.732928 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:09.723148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:42795 (local address 127.25.240.67:44511)
0924 13:40:09.723667 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:09.723712 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:09.724224 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:09.724463 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:09.724467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:09.724657 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:09.724849 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:09.724853 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.725546 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.725548 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:09.726408 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:09.726410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:09.726896 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:09.726897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:09.727038 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:09.727591 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:09.727601 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:09.728936 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:09.730772 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:09.730778 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:09.730780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:09.731088 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:09.731310 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:09.731312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:09.731312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:09.731451 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:09.731689 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:09.731695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:09.731840 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 13:40:09.732511 (+ 671us) client_negotiation.cc:240] Negotiation successful
0924 13:40:09.732674 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":434,"threads_started":1}
I20260924 13:40:09.733543 6796 heartbeater.cc:347] Connected to a master server at 127.25.240.126:42795
I20260924 13:40:09.733757 6796 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:09.733997 6796 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:09.734393 6312 ts_manager.cc:194] Registered new tserver with Master: eb078d61dfd443e693e67d3c1f1ece2c (127.25.240.67:44641)
I20260924 13:40:09.734836 6312 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:44511
I20260924 13:40:09.742187 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:40:09.742746 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 6393
I20260924 13:40:09.747870 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 6532
I20260924 13:40:09.752931 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 6667
I20260924 13:40:09.756814 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 6280
I20260924 13:40:09.761129 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:42795
--webserver_interface=127.25.240.126
--webserver_port=41471
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:42795
--ranger_config_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:33903/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.240.84:33903/kms with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:09.840296 6799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:09.840490 6799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:09.840519 6799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:09.841651 6799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:09.841691 6799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:09.841704 6799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:09.841717 6799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:09.841729 6799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:09.843266 6799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.240.84:33903/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:42795
--ranger_config_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=41471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:09.843442 6799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:09.843603 6799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:09.848260 6804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:09.848290 6805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:09.848260 6807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:09.848495 6799 server_base.cc:1034] running on GCE node
I20260924 13:40:09.848803 6799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:09.849041 6799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:09.850205 6799 hybrid_clock.cc:648] HybridClock initialized: now 1790257209850190 us; error 34 us; skew 500 ppm
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:09.852447 6799 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:09.852850 6799 webserver.cc:460] Webserver started at http://127.25.240.126:41471/ using document root <none> and password file <none>
I20260924 13:40:09.853065 6799 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:09.853135 6799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:40:09 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257209, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:09.917587 6799 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:09.918540 6815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:09.918744 6799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:09.918848 6799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "9f090c2ea20c482e87db7a121244e77a"
format_stamp: "Formatted at 2026-09-24 13:40:08 on dist-test-slave-m5z1"
server_key: "68e6c9adf012181ca2b485983af337fb"
server_key_iv: "9b9b1af7e758de12e6729e319a826802"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:09.919322 6799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:09.946842 6799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:09.948473 6799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:09.948652 6799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:09.952816 6799 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:42795
I20260924 13:40:09.952859 6867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:42795 every 8 connection(s)
I20260924 13:40:09.953233 6799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:09.954478 6868 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:40:09.955128 6868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Bootstrap starting.
I20260924 13:40:09.957360 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 6799
I20260924 13:40:09.957696 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:34009
--local_ip_for_outbound_sockets=127.25.240.65
--tserver_master_addrs=127.25.240.126:42795
--webserver_port=45633
--webserver_interface=127.25.240.65
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.240.84:33903/kms with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:09.963696 6868 log.cc:824] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:09.965322 6868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:40:09.965667 6868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Bootstrap complete.
I20260924 13:40:09.968024 6868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } }
I20260924 13:40:09.968256 6868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f090c2ea20c482e87db7a121244e77a, State: Initialized, Role: FOLLOWER
I20260924 13:40:09.968420 6868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } }
I20260924 13:40:09.968490 6868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:40:09.968549 6868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:40:09.968598 6868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:40:09.969645 6868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } }
I20260924 13:40:09.969756 6868 leader_election.cc:302] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f090c2ea20c482e87db7a121244e77a; no voters:
I20260924 13:40:09.969909 6868 leader_election.cc:288] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:40:09.969945 6872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:40:09.970103 6872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [term 2 LEADER]: Becoming Leader. State: Replica: 9f090c2ea20c482e87db7a121244e77a, State: Running, Role: LEADER
I20260924 13:40:09.970172 6868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:09.970247 6872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } }
I20260924 13:40:09.970584 6872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f090c2ea20c482e87db7a121244e77a. Latest consensus state: current_term: 2 leader_uuid: "9f090c2ea20c482e87db7a121244e77a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } } }
I20260924 13:40:09.970587 6874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9f090c2ea20c482e87db7a121244e77a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f090c2ea20c482e87db7a121244e77a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 42795 } } }
I20260924 13:40:09.970844 6874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:09.970759 6872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:09.971895 6881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:09.972198 6868 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 13:40:09.972399 6881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:09.972541 6881 catalog_manager.cc:1259] Loaded cluster ID: 481adfa8a5f64f88ad1ffdcef435fbc7
I20260924 13:40:09.972579 6881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:40:09.973398 6881 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:40:09.973522 6881 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Loaded TSK: 0
I20260924 13:40:09.973768 6881 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:40:10.068704 6870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:10.068984 6870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:10.069021 6870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:10.070781 6870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:10.070838 6870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:10.070879 6870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:40:10.072938 6870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.240.84:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:34009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=45633
--tserver_master_addrs=127.25.240.126:42795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:10.073335 6870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:10.073668 6870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:10.080339 6906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:10.080910 6905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:10.081159 6908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:10.081312 6870 server_base.cc:1034] running on GCE node
I20260924 13:40:10.081650 6870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:10.081923 6870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:10.083098 6870 hybrid_clock.cc:648] HybridClock initialized: now 1790257210083076 us; error 35 us; skew 500 ppm
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:10.086347 6870 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:40:10.086870 6870 webserver.cc:460] Webserver started at http://127.25.240.65:45633/ using document root <none> and password file <none>
I20260924 13:40:10.087147 6870 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:10.087239 6870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:10.105975 6870 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 13:40:10.107697 6916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.107911 6870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260924 13:40:10.107999 6870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "5f3e33afc05c4a208d39b668ec976490"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "aba4dd66d379b3363fcf7ff2df5520e0"
server_key_iv: "8b9a9533900fb8473f5637968ee6e357"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:10.108479 6870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:10.127993 6870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:10.130591 6870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:10.130815 6870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:10.131106 6870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:10.131512 6870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:10.131609 6870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.131713 6870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:10.131786 6870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.139227 6870 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:34009
I20260924 13:40:10.139751 6870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:40:10.141458 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:34009 every 8 connection(s)
I20260924 13:40:10.143718 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 6870
I20260924 13:40:10.144002 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:45305
--local_ip_for_outbound_sockets=127.25.240.66
--tserver_master_addrs=127.25.240.126:42795
--webserver_port=35249
--webserver_interface=127.25.240.66
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.240.84:33903/kms with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:10.171679 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:10.142288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:42795 (local address 127.25.240.65:38339)
0924 13:40:10.142749 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:10.142792 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:10.143389 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:10.150310 (+ 6921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:10.150314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:10.150607 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:10.150893 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:10.150900 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.153242 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.153244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:10.154551 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:10.154555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.165532 (+ 10977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.165534 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:10.165703 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:10.166361 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:10.166371 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:10.168039 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:10.169989 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:10.169996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:10.169998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:10.170244 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:10.170488 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:10.170489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:10.170490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:10.170605 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:10.171003 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:10.171009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:10.171143 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 13:40:10.171291 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 13:40:10.171457 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":93,"threads_started":1}
I20260924 13:40:10.171677 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:10.142335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:38339 (local address 127.25.240.126:42795)
0924 13:40:10.149702 (+ 7367us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:10.149747 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:10.149791 (+ 44us) server_negotiation.cc:407] Connection header received
0924 13:40:10.149977 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:10.149984 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:10.150044 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:10.150194 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:10.151095 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.153075 (+ 1980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.155468 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.165274 (+ 9806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.165400 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:10.168162 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:10.168167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:10.168167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:10.168179 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:10.169899 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:10.170325 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:10.170329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:10.170329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:10.170417 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:10.170678 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:10.170679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:10.170679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:10.170940 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:10.171006 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:10.171342 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 13:40:10.171457 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7307,"thread_start_us":71,"threads_started":1}
I20260924 13:40:10.172834 7030 heartbeater.cc:347] Connected to a master server at 127.25.240.126:42795
I20260924 13:40:10.172931 7030 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.173085 7030 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
W20260924 13:40:10.173902 7030 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:42795 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 13:40:10.173981 7030 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.174045 7030 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:10.174355 7030 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.174429 7030 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
W20260924 13:40:10.174711 7030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:40:10.268623 7034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:10.268903 7034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:10.268944 7034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:10.270555 7034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:10.270620 7034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:10.270661 7034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:40:10.272267 7034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.240.84:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:45305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=35249
--tserver_master_addrs=127.25.240.126:42795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:10.272504 7034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:10.272732 7034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:10.277935 7042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:10.277967 7039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:10.278163 7040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:10.278581 7034 server_base.cc:1034] running on GCE node
I20260924 13:40:10.278782 7034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:10.279055 7034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:10.280237 7034 hybrid_clock.cc:648] HybridClock initialized: now 1790257210280209 us; error 44 us; skew 500 ppm
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:10.283643 7034 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:40:10.284161 7034 webserver.cc:460] Webserver started at http://127.25.240.66:35249/ using document root <none> and password file <none>
I20260924 13:40:10.284432 7034 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:10.284519 7034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:10.298688 7034 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 13:40:10.303103 7050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.303325 7034 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260924 13:40:10.303473 7034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "7243f028684b4c1d9cd7f24cd56db6e0"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "c58aac20966d3f7726197400eb577412"
server_key_iv: "15c18ae38df971c3bdd49520df0dc168"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:10.303845 7034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:10.317140 7034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:10.319208 7034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:10.319388 7034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:10.319675 7034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:10.320050 7034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:10.320101 7034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.320142 7034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:10.320169 7034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.327621 7034 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:45305
I20260924 13:40:10.328078 7034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:40:10.328300 7163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:45305 every 8 connection(s)
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:10.335364 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 7034
I20260924 13:40:10.335707 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.240.84:33903/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:44641
--local_ip_for_outbound_sockets=127.25.240.67
--tserver_master_addrs=127.25.240.126:42795
--webserver_port=45785
--webserver_interface=127.25.240.67
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.240.84:33903/kms with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:10.340279 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:10.329046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:54277 (local address 127.25.240.126:42795)
0924 13:40:10.329103 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:10.329105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:10.329311 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:40:10.329928 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:10.329929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:10.329941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:10.329996 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:10.330539 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.331042 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.332030 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.332254 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.332318 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:10.334802 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:10.334803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:10.334804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:10.334811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:10.338546 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:10.339082 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:10.339083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:10.339084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:10.339149 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:10.339464 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:10.339465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:10.339466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:10.339631 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:10.339710 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:10.340089 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 13:40:10.340186 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 13:40:10.340812 7166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:10.329035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:42795 (local address 127.25.240.66:54277)
0924 13:40:10.329197 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:10.329232 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:10.329823 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:10.330066 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:10.330069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:10.330243 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:10.330437 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:10.330441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.331136 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.331137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:10.331931 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:10.331933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.332359 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.332361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:10.332526 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:10.333038 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:10.333048 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:10.334688 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:10.338643 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:10.338650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:10.338653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:10.338983 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:10.339225 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:10.339226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:10.339227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:10.339380 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:10.339714 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:10.339721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:10.339904 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 13:40:10.340389 (+ 485us) client_negotiation.cc:240] Negotiation successful
0924 13:40:10.340580 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260924 13:40:10.341343 7164 heartbeater.cc:347] Connected to a master server at 127.25.240.126:42795
I20260924 13:40:10.341445 7164 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.341681 7164 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
W20260924 13:40:10.342425 7164 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:42795 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 13:40:10.342494 7164 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.342553 7164 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:10.342929 7164 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.342999 7164 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
W20260924 13:40:10.343312 7164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:40:10.436542 7167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:10.436834 7167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:10.436870 7167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:10.438551 7167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:10.438627 7167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:10.438666 7167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:40:10.440770 7167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.240.84:33903/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:44641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=45785
--tserver_master_addrs=127.25.240.126:42795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:10.441022 7167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:10.441303 7167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:40:10.447001 7167 server_base.cc:1034] running on GCE node
W20260924 13:40:10.449030 7176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:10.449132 7174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:10.449030 7173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:10.449486 7167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:10.449759 7167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:10.450939 7167 hybrid_clock.cc:648] HybridClock initialized: now 1790257210450929 us; error 30 us; skew 500 ppm
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:10.453393 7167 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:40:10.453908 7167 webserver.cc:460] Webserver started at http://127.25.240.67:45785/ using document root <none> and password file <none>
I20260924 13:40:10.454149 7167 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:10.454224 7167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:10.477738 7167 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 13:40:10.480127 7184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.480333 7167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260924 13:40:10.480424 7167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "eb078d61dfd443e693e67d3c1f1ece2c"
format_stamp: "Formatted at 2026-09-24 13:40:09 on dist-test-slave-m5z1"
server_key: "57934863edc2e6a349b4d86537aa8f3f"
server_key_iv: "465505b6eebd23daed4cf40084c1cd79"
server_key_version: "kuduclusterkey@0"
I20260924 13:40:10.480906 7167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:10.492520 7167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:10.494576 7167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:10.494735 7167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:10.494966 7167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:10.495281 7167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:10.495332 7167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.495373 7167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:10.495399 7167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:10.501055 7167 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:44641
I20260924 13:40:10.501174 7297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:44641 every 8 connection(s)
I20260924 13:40:10.501466 7167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:40:10.501742 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 7167
Sep 24 13:40:10 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257210, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:10.509966 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:10.502191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.25.240.126:42795)
0924 13:40:10.502312 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:10.502314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:10.502531 (+ 217us) server_negotiation.cc:407] Connection header received
0924 13:40:10.502569 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:10.502570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:10.502583 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:10.502653 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:10.503434 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.504217 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.505171 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.505466 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.505601 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:10.506962 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:10.506963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:10.506964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:10.506971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:10.508626 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:10.509057 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:10.509059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:10.509059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:10.509120 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:10.509351 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:10.509352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:10.509352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:10.509494 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:10.509693 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:10.509783 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 13:40:10.509866 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 13:40:10.512619 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:10.502210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:51251 (local address 127.25.240.126:42795)
0924 13:40:10.502606 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:10.502608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:10.502828 (+ 220us) server_negotiation.cc:407] Connection header received
0924 13:40:10.503498 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:10.503499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:10.503514 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:10.503579 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:10.504097 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.504815 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.506793 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.507131 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.507471 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:10.509402 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:10.509403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:10.509403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:10.509410 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:10.511137 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:10.511529 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:10.511530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:10.511530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:10.511594 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:10.511888 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:10.511889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:10.511889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:10.512010 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:10.512054 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:10.512424 (+ 370us) server_negotiation.cc:299] Negotiation successful
0924 13:40:10.512507 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":57,"threads_started":1}
I20260924 13:40:10.513115 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:10.502202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:42795 (local address 127.25.240.67:51251)
0924 13:40:10.502688 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:10.502739 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:10.503413 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:10.503648 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:10.503653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:10.503818 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:10.504002 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:10.504006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.504909 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.504911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:10.506706 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:10.506709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:10.507239 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:10.507241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:10.507387 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:10.507880 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:10.507890 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:10.509294 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:10.511219 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:10.511226 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:10.511228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:10.511442 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:10.511683 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:10.511684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:10.511684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:10.511811 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:10.512103 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:10.512109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:10.512265 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 13:40:10.512739 (+ 474us) client_negotiation.cc:240] Negotiation successful
0924 13:40:10.512893 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":80,"threads_started":1}
I20260924 13:40:10.514907 7298 heartbeater.cc:347] Connected to a master server at 127.25.240.126:42795
I20260924 13:40:10.515012 7298 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.515203 7298 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
W20260924 13:40:10.515964 7298 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:42795 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 13:40:10.516038 7298 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.516099 7298 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:10.516474 7298 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:10.516561 7298 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
W20260924 13:40:10.516914 7298 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 13:40:11 dist-test-slave-m5z1 krb5kdc[5836](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790257211, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:11.174904 7030 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:11.175068 7030 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:11.175480 6832 ts_manager.cc:194] Registered new tserver with Master: 5f3e33afc05c4a208d39b668ec976490 (127.25.240.65:34009)
I20260924 13:40:11.176519 6832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:38339
I20260924 13:40:11.343533 7164 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:11.343701 7164 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:11.344043 6832 ts_manager.cc:194] Registered new tserver with Master: 7243f028684b4c1d9cd7f24cd56db6e0 (127.25.240.66:45305)
I20260924 13:40:11.344452 6832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:54277
Sep 24 13:40:11 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257211, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:11.517098 7298 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:11.517262 7298 heartbeater.cc:511] Master 127.25.240.126:42795 requested a full tablet report, sending...
I20260924 13:40:11.517748 6832 ts_manager.cc:194] Registered new tserver with Master: eb078d61dfd443e693e67d3c1f1ece2c (127.25.240.67:44641)
I20260924 13:40:11.518332 6832 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:51251
I20260924 13:40:11.521163 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:40:11.528795 7320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.522138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.25.240.126:42795)
0924 13:40:11.522385 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:11.522386 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:11.522392 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:40:11.522409 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:11.522409 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:11.522417 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:11.522490 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:11.523083 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.523686 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.524592 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.524808 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.524889 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.526168 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:11.526169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:11.526170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:11.526177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:11.527666 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:11.528011 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:11.528012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:11.528013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:11.528072 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:11.528270 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:11.528271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:11.528271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:11.528376 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:11.528479 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:11.528590 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 13:40:11.528672 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":72,"threads_started":1}
I20260924 13:40:11.530696 6832 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:40:11.532493 6832 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:40:11.540556 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.535703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:44641 (local address 127.0.0.1:55420)
0924 13:40:11.536021 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:11.536030 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:11.536133 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:11.536441 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:11.536441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:11.536443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:11.536515 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.536516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.537801 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.537802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.539145 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.539147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.540201 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.540202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.540282 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.540299 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 13:40:11.540365 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:40:11.540416 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":4,"thread_start_us":81,"threads_started":1}
I20260924 13:40:11.540560 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.535084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:34009 (local address 127.0.0.1:55212)
0924 13:40:11.535217 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:11.535256 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:11.535315 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:11.536424 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:11.536427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:11.536429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:11.536518 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.536519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.538053 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.538054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.539144 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.539147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.540382 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.540383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.540442 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.540445 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:40:11.540474 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 13:40:11.540502 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":25,"thread_start_us":82,"threads_started":1}
I20260924 13:40:11.541064 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.535793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.25.240.67:44641)
0924 13:40:11.535931 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:11.535937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:11.536110 (+ 173us) server_negotiation.cc:407] Connection header received
0924 13:40:11.536264 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:11.536267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:11.536286 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:11.536361 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:11.536595 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.537719 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.539247 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.540005 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.540097 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.540166 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:11.540919 (+ 753us) server_negotiation.cc:299] Negotiation successful
0924 13:40:11.540968 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":51,"threads_started":1}
I20260924 13:40:11.541352 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.535494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.25.240.65:34009)
0924 13:40:11.536261 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:11.536266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:11.536274 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:40:11.536304 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:11.536306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:11.536322 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:11.536594 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:11.536662 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.537684 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.539249 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.540001 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.540650 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.540728 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:11.540756 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 13:40:11.540797 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":51,"threads_started":1}
I20260924 13:40:11.541894 7232 tablet_service.cc:1467] Processing CreateTablet for tablet fa430b786bac40f5a9fe7968280d219a (DEFAULT_TABLE table=test-table [id=ffbc1b990f4a4dc3bf67df1090c59854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:11.542079 6964 tablet_service.cc:1467] Processing CreateTablet for tablet fa430b786bac40f5a9fe7968280d219a (DEFAULT_TABLE table=test-table [id=ffbc1b990f4a4dc3bf67df1090c59854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:11.542335 7232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa430b786bac40f5a9fe7968280d219a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:11.542411 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa430b786bac40f5a9fe7968280d219a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:11.545444 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.535333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:45305 (local address 127.0.0.1:48888)
0924 13:40:11.535458 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:11.535469 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:11.535509 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:11.541852 (+ 6343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:11.541853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:11.541855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:11.541910 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.541912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.543159 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.543160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.544242 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.544245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.545221 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.545222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.545300 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.545305 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:40:11.545337 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 13:40:11.545373 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":89,"threads_started":1}
I20260924 13:40:11.545821 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.535444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48888 (local address 127.25.240.66:45305)
0924 13:40:11.541677 (+ 6233us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:11.541682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:11.541693 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:40:11.541730 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:11.541732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:11.541748 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:11.541982 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:11.542060 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.543044 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.544351 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.545117 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.545592 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.545659 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:11.545687 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 13:40:11.545729 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6192,"thread_start_us":58,"threads_started":1}
I20260924 13:40:11.546550 7093 tablet_service.cc:1467] Processing CreateTablet for tablet fa430b786bac40f5a9fe7968280d219a (DEFAULT_TABLE table=test-table [id=ffbc1b990f4a4dc3bf67df1090c59854]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:11.546885 7093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa430b786bac40f5a9fe7968280d219a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:11.550067 7332 tablet_bootstrap.cc:492] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490: Bootstrap starting.
I20260924 13:40:11.550806 7332 tablet_bootstrap.cc:654] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:11.549958 7333 tablet_bootstrap.cc:492] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Bootstrap starting.
I20260924 13:40:11.551174 7332 log.cc:824] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:11.551668 7333 tablet_bootstrap.cc:654] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:11.551939 7333 log.cc:824] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:11.552649 7333 tablet_bootstrap.cc:492] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: No bootstrap required, opened a new log
I20260924 13:40:11.552731 7333 ts_tablet_manager.cc:1397] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 13:40:11.554035 7332 tablet_bootstrap.cc:492] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490: No bootstrap required, opened a new log
I20260924 13:40:11.554183 7332 ts_tablet_manager.cc:1397] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 13:40:11.555351 7333 raft_consensus.cc:348] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.555857 7333 raft_consensus.cc:374] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:11.555966 7333 raft_consensus.cc:729] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7243f028684b4c1d9cd7f24cd56db6e0, State: Initialized, Role: FOLLOWER
I20260924 13:40:11.556175 7333 consensus_queue.cc:260] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.556538 7333 ts_tablet_manager.cc:1428] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 13:40:11.556735 7164 heartbeater.cc:503] Master 127.25.240.126:42795 was elected leader, sending a full tablet report...
I20260924 13:40:11.556908 7332 raft_consensus.cc:348] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.557056 7332 raft_consensus.cc:374] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:11.557106 7332 raft_consensus.cc:729] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3e33afc05c4a208d39b668ec976490, State: Initialized, Role: FOLLOWER
I20260924 13:40:11.557214 7332 consensus_queue.cc:260] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.557364 7331 tablet_bootstrap.cc:492] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Bootstrap starting.
I20260924 13:40:11.557449 7332 ts_tablet_manager.cc:1428] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:40:11.557576 7030 heartbeater.cc:503] Master 127.25.240.126:42795 was elected leader, sending a full tablet report...
I20260924 13:40:11.558655 7331 tablet_bootstrap.cc:654] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:11.558961 7331 log.cc:824] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:11.559752 7331 tablet_bootstrap.cc:492] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: No bootstrap required, opened a new log
I20260924 13:40:11.559825 7331 ts_tablet_manager.cc:1397] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 13:40:11.562418 7331 raft_consensus.cc:348] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.562587 7331 raft_consensus.cc:374] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:11.562618 7331 raft_consensus.cc:729] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb078d61dfd443e693e67d3c1f1ece2c, State: Initialized, Role: FOLLOWER
I20260924 13:40:11.562775 7331 consensus_queue.cc:260] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.563024 7331 ts_tablet_manager.cc:1428] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:40:11.563153 7298 heartbeater.cc:503] Master 127.25.240.126:42795 was elected leader, sending a full tablet report...
W20260924 13:40:11.579423 7165 tablet.cc:2369] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:40:11.643729 7031 tablet.cc:2369] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 13:40:11 dist-test-slave-m5z1 krb5kdc[5836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257211, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for HTTP/127.25.240.84@KRBTEST.COM
I20260924 13:40:11.711472 7339 raft_consensus.cc:482] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:11.711640 7339 raft_consensus.cc:504] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.712102 7339 leader_election.cc:288] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f3e33afc05c4a208d39b668ec976490 (127.25.240.65:34009), 7243f028684b4c1d9cd7f24cd56db6e0 (127.25.240.66:45305)
I20260924 13:40:11.721257 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.712779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:49285 (local address 127.25.240.65:34009)
0924 13:40:11.712826 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:11.712828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:11.716524 (+ 3696us) server_negotiation.cc:407] Connection header received
0924 13:40:11.716636 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:11.716637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:11.716647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:11.716679 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:11.716899 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.717599 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.719840 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.720329 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.720368 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.720376 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:11.721126 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 13:40:11.721177 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:40:11.721902 7343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.712361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:34009 (local address 127.25.240.67:49285)
0924 13:40:11.716458 (+ 4097us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:11.716471 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:11.716557 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:11.716748 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:11.716749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:11.716750 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:11.716827 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.716828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.718047 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.718048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.719730 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.719734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.720914 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.720915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.720983 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.720988 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:40:11.721733 (+ 745us) client_negotiation.cc:240] Negotiation successful
0924 13:40:11.721792 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4076,"thread_start_us":98,"threads_started":1}
I20260924 13:40:11.723963 6981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa430b786bac40f5a9fe7968280d219a" candidate_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3e33afc05c4a208d39b668ec976490" is_pre_election: true
I20260924 13:40:11.724121 6981 raft_consensus.cc:2454] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb078d61dfd443e693e67d3c1f1ece2c in term 0.
I20260924 13:40:11.724438 7187 leader_election.cc:302] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3e33afc05c4a208d39b668ec976490, eb078d61dfd443e693e67d3c1f1ece2c; no voters:
I20260924 13:40:11.724565 7339 raft_consensus.cc:2781] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:40:11.724641 7339 raft_consensus.cc:482] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:40:11.724673 7339 raft_consensus.cc:3037] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:11.725580 7339 raft_consensus.cc:504] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.725720 7339 leader_election.cc:288] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [CANDIDATE]: Term 1 election: Requested vote from peers 5f3e33afc05c4a208d39b668ec976490 (127.25.240.65:34009), 7243f028684b4c1d9cd7f24cd56db6e0 (127.25.240.66:45305)
I20260924 13:40:11.725901 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.712908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:45305 (local address 127.25.240.67:37347)
0924 13:40:11.721413 (+ 8505us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:11.721425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:11.721517 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:11.721716 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:11.721716 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:11.721718 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:11.721779 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.721781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.722626 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.722627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.723751 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:11.723755 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.724499 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.724500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:11.725700 (+ 1200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.725706 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:40:11.725784 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 13:40:11.725822 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8486,"thread_start_us":88,"threads_started":1}
I20260924 13:40:11.725919 6981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa430b786bac40f5a9fe7968280d219a" candidate_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3e33afc05c4a208d39b668ec976490"
I20260924 13:40:11.725987 6981 raft_consensus.cc:3037] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:11.726789 6981 raft_consensus.cc:2454] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb078d61dfd443e693e67d3c1f1ece2c in term 1.
I20260924 13:40:11.727047 7187 leader_election.cc:302] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3e33afc05c4a208d39b668ec976490, eb078d61dfd443e693e67d3c1f1ece2c; no voters:
I20260924 13:40:11.727126 7339 raft_consensus.cc:2781] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:11.727269 7339 raft_consensus.cc:686] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 1 LEADER]: Becoming Leader. State: Replica: eb078d61dfd443e693e67d3c1f1ece2c, State: Running, Role: LEADER
I20260924 13:40:11.727360 7339 consensus_queue.cc:237] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } }
I20260924 13:40:11.727978 7327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.712859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:37347 (local address 127.25.240.66:45305)
0924 13:40:11.713390 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:11.713392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:11.721489 (+ 8097us) server_negotiation.cc:407] Connection header received
0924 13:40:11.721617 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:11.721618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:11.721628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:11.721656 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:11.721864 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.722528 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.723843 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.724429 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.724500 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.724508 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:11.727851 (+ 3343us) server_negotiation.cc:299] Negotiation successful
0924 13:40:11.727912 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494}
I20260924 13:40:11.728139 6832 catalog_manager.cc:5495] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c reported cstate change: term changed from 0 to 1, leader changed from <none> to eb078d61dfd443e693e67d3c1f1ece2c (127.25.240.67). New cstate: current_term: 1 leader_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44641 } health_report { overall_health: HEALTHY } } }
I20260924 13:40:11.728711 7116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa430b786bac40f5a9fe7968280d219a" candidate_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7243f028684b4c1d9cd7f24cd56db6e0"
I20260924 13:40:11.728838 7116 raft_consensus.cc:3037] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:11.728711 7118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa430b786bac40f5a9fe7968280d219a" candidate_uuid: "eb078d61dfd443e693e67d3c1f1ece2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" is_pre_election: true
I20260924 13:40:11.729733 7116 raft_consensus.cc:2454] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb078d61dfd443e693e67d3c1f1ece2c in term 1.
W20260924 13:40:11.756605 7299 tablet.cc:2369] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:40:11.771409 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.767936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.25.240.67:44641)
0924 13:40:11.767982 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:11.767984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:11.767991 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:40:11.768046 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:11.768047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:11.768069 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:11.768139 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:40:11.768331 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.769069 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.770211 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:11.770503 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:11.770574 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:11.770808 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:40:11.770943 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:40:11.771040 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:11.771306 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 13:40:11.771328 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:40:11.780288 7116 raft_consensus.cc:1261] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [term 1 FOLLOWER]: Refusing update from remote peer eb078d61dfd443e693e67d3c1f1ece2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:11.780637 6981 raft_consensus.cc:1261] T fa430b786bac40f5a9fe7968280d219a P 5f3e33afc05c4a208d39b668ec976490 [term 1 FOLLOWER]: Refusing update from remote peer eb078d61dfd443e693e67d3c1f1ece2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:11.781086 7344 consensus_queue.cc:1035] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3e33afc05c4a208d39b668ec976490" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 34009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:11.781270 7344 consensus_queue.cc:1035] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7243f028684b4c1d9cd7f24cd56db6e0" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 45305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:11.793303 7347 mvcc.cc:205] Tried to move back new op lower bound from 7332893539448676352 to 7332893539235815424. Current Snapshot: MvccSnapshot[applied={T|T < 7332893539448676352}]
I20260924 13:40:11.795049 7350 mvcc.cc:205] Tried to move back new op lower bound from 7332893539448676352 to 7332893539235815424. Current Snapshot: MvccSnapshot[applied={T|T < 7332893539448676352}]
I20260924 13:40:11.798305 6832 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:11.798419 6832 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:11.799160 6832 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f090c2ea20c482e87db7a121244e77a: Sending DeleteTablet for 3 replicas of tablet fa430b786bac40f5a9fe7968280d219a
I20260924 13:40:11.799620 6964 tablet_service.cc:1514] Processing DeleteTablet for tablet fa430b786bac40f5a9fe7968280d219a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:11 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:55212
I20260924 13:40:11.799880 7093 tablet_service.cc:1514] Processing DeleteTablet for tablet fa430b786bac40f5a9fe7968280d219a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:11 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:48888
I20260924 13:40:11.800311 7232 tablet_service.cc:1514] Processing DeleteTablet for tablet fa430b786bac40f5a9fe7968280d219a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:11 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:55420
I20260924 13:40:11.800806 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 6870
I20260924 13:40:11.802512 7358 tablet_replica.cc:330] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: stopping tablet replica
I20260924 13:40:11.802620 7358 raft_consensus.cc:2229] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:40:11.802740 7358 raft_consensus.cc:2258] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:11.803194 7358 ts_tablet_manager.cc:1905] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:11.805958 7357 tablet_replica.cc:330] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: stopping tablet replica
I20260924 13:40:11.806095 7357 raft_consensus.cc:2229] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:11.806218 7357 raft_consensus.cc:2258] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:11.806711 7357 ts_tablet_manager.cc:1905] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:11.809427 7358 ts_tablet_manager.cc:1918] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:11.809516 7358 log.cc:1196] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/fa430b786bac40f5a9fe7968280d219a
I20260924 13:40:11.809823 7358 ts_tablet_manager.cc:1939] T fa430b786bac40f5a9fe7968280d219a P eb078d61dfd443e693e67d3c1f1ece2c: Deleting consensus metadata
I20260924 13:40:11.810276 6817 catalog_manager.cc:4841] TS eb078d61dfd443e693e67d3c1f1ece2c (127.25.240.67:44641): tablet fa430b786bac40f5a9fe7968280d219a (table test-table [id=ffbc1b990f4a4dc3bf67df1090c59854]) successfully deleted
I20260924 13:40:11.810622 7357 ts_tablet_manager.cc:1918] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 13:40:11.813182 6818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:34009 (error 108)
I20260924 13:40:11.813355 7357 log.cc:1196] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/fa430b786bac40f5a9fe7968280d219a
I20260924 13:40:11.813766 7357 ts_tablet_manager.cc:1939] T fa430b786bac40f5a9fe7968280d219a P 7243f028684b4c1d9cd7f24cd56db6e0: Deleting consensus metadata
I20260924 13:40:11.814244 6816 catalog_manager.cc:4841] TS 7243f028684b4c1d9cd7f24cd56db6e0 (127.25.240.66:45305): tablet fa430b786bac40f5a9fe7968280d219a (table test-table [id=ffbc1b990f4a4dc3bf67df1090c59854]) successfully deleted
I20260924 13:40:11.821704 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 7034
I20260924 13:40:11.823593 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:11.823457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:34009 (local address 127.0.0.1:55214)
0924 13:40:11.823497 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:11.823554 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:34009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 13:40:11.823756 6818 catalog_manager.cc:4584] TS 5f3e33afc05c4a208d39b668ec976490 (127.25.240.65:34009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa430b786bac40f5a9fe7968280d219a: Network error: Client connection negotiation failed: client connection to 127.25.240.65:34009: connect: Connection refused (error 111)
I20260924 13:40:11.827760 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 7167
I20260924 13:40:11.835269 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 6799
I20260924 13:40:11.842418 26561 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 13:40:12.186748 26561 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 13:40:12.186836 26561 mini_ranger.cc:66] Stopping Ranger...
I20260924 13:40:12.531144 26561 mini_ranger.cc:68] Stopped Ranger
2026-09-24 13:40:12.531 UTC [5866] LOG: received smart shutdown request
2026-09-24 13:40:12.535 UTC [5866] LOG: background worker "logical replication launcher" (PID 5875) exited with exit code 1
2026-09-24 13:40:12.535 UTC [5870] LOG: shutting down
2026-09-24 13:40:12.707 UTC [5866] LOG: database system is shut down
2026-09-24T13:40:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47399 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:40:12 dist-test-slave-m5z1 krb5kdc[7368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:40:12 dist-test-slave-m5z1 krb5kdc[7368](info): set up 2 sockets
Sep 24 13:40:12 dist-test-slave-m5z1 krb5kdc[7368](info): commencing operation
krb5kdc: starting...
W20260924 13:40:13.865880 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:40:13 dist-test-slave-m5z1 krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:40:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:40:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:40:13.982776 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:36877
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:36877,127.25.240.125:40935,127.25.240.124:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:14.062086 7381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:14.062284 7381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:14.062304 7381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:14.063372 7381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:14.063409 7381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:14.063431 7381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:14.063443 7381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:14.063455 7381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:14.064966 7381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:36877,127.25.240.125:40935,127.25.240.124:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:14.065153 7381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:14.065310 7381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:14.069983 7389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:14.070036 7381 server_base.cc:1034] running on GCE node
W20260924 13:40:14.069972 7387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:14.069985 7386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:14.070376 7381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:14.070672 7381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:14.071835 7381 hybrid_clock.cc:648] HybridClock initialized: now 1790257214071819 us; error 28 us; skew 500 ppm
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:14.074237 7381 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:14.074677 7381 webserver.cc:460] Webserver started at http://127.25.240.126:35955/ using document root <none> and password file <none>
I20260924 13:40:14.074855 7381 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:14.074895 7381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:14.074981 7381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:14.075937 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "f631c23fb110f9e0d097a1bc30111645"
server_key_iv: "0fd52acff2bbb088bd886df24c70c8b9"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.076222 7381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "f631c23fb110f9e0d097a1bc30111645"
server_key_iv: "0fd52acff2bbb088bd886df24c70c8b9"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.077458 7381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:40:14.078213 7396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:14.078423 7381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:14.078544 7381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "f631c23fb110f9e0d097a1bc30111645"
server_key_iv: "0fd52acff2bbb088bd886df24c70c8b9"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.078619 7381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:14.094686 7381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:14.096380 7381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:14.096531 7381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:14.100203 7381 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:36877
I20260924 13:40:14.100232 7448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:36877 every 8 connection(s)
I20260924 13:40:14.100565 7381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:14.101385 7449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:14.102916 7449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:40:14.107448 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 7381
I20260924 13:40:14.107561 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:40:14.107767 26561 external_mini_cluster.cc:1428] Setting key dc1be8159b3ad3cafabd8b961a3b3c6f
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:14.112578 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.103572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:36877 (local address 127.0.0.1:33366)
0924 13:40:14.103738 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.103774 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.104202 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.104429 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.104431 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.104673 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.104826 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.104830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.105855 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.105856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.106510 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.106512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.106953 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.106954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.107010 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.107462 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.107473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.108952 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.110985 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.110991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.110993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.111204 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.111592 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.111594 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.111594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.111707 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.112162 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.112167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.112223 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.112295 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.112427 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260924 13:40:14.112699 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.103590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.25.240.126:36877)
0924 13:40:14.103760 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.103769 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.103861 (+ 92us) server_negotiation.cc:407] Connection header received
0924 13:40:14.104320 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.104322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.104337 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.104415 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.104904 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.105777 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.106583 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.106879 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.106970 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.109051 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.109057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.109057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.109063 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.110908 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.111289 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.111292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.111293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.111398 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.111798 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.111800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.111800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.112088 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.112178 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.112352 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.112519 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260924 13:40:14.113348 7449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:40:14.113706 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.113611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60316)
0924 13:40:14.113641 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.113681 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 13:40:14.113826 7397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
I20260924 13:40:14.114202 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.114111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60318)
0924 13:40:14.114159 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.114182 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 13:40:14.114296 7449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:40935: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.25.240.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab.
Entry for principal kudu/127.25.240.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab.
I20260924 13:40:14.132411 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.125:40935
--webserver_interface=127.25.240.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab
--principal=kudu/127.25.240.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:36877,127.25.240.125:40935,127.25.240.124:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:14.163504 7449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } attempt: 1
I20260924 13:40:14.164023 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.163866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60332)
0924 13:40:14.163935 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.163985 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 13:40:14.164420 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.164317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60338)
0924 13:40:14.164368 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.164401 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 13:40:14.164588 7449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:40935: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
W20260924 13:40:14.213092 7456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:14.213289 7456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:14.213311 7456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:14.214511 7456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:14.214553 7456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:14.214567 7456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:14.214602 7456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:14.214629 7456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:14.216069 7456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:36877,127.25.240.125:40935,127.25.240.124:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab
--principal=kudu/127.25.240.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.125:40935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.240.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:14.216313 7456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:14.216532 7456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:14.221215 7462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:14.221377 7461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:14.221370 7456 server_base.cc:1034] running on GCE node
W20260924 13:40:14.221369 7464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:14.221760 7456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:14.222008 7456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:14.223176 7456 hybrid_clock.cc:648] HybridClock initialized: now 1790257214223153 us; error 39 us; skew 500 ppm
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:14.225934 7456 init.cc:377] Logged in from keytab as kudu/127.25.240.125@KRBTEST.COM (short username kudu)
I20260924 13:40:14.226413 7456 webserver.cc:460] Webserver started at http://127.25.240.125:40225/ using document root <none> and password file <none>
I20260924 13:40:14.226636 7456 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:14.226702 7456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:14.226852 7456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:14.227926 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/data/instance:
uuid: "28a4f2fd1da54642aa7436a3f095bca3"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "8e3de797c57f1ee14c0cb3087ffd0ab2"
server_key_iv: "a1e35137f44a387098ccc5ab7a05be1c"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.228293 7456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/wal/instance:
uuid: "28a4f2fd1da54642aa7436a3f095bca3"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "8e3de797c57f1ee14c0cb3087ffd0ab2"
server_key_iv: "a1e35137f44a387098ccc5ab7a05be1c"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.229627 7456 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:40:14.230345 7471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:14.230546 7456 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:14.230651 7456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/wal
uuid: "28a4f2fd1da54642aa7436a3f095bca3"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "8e3de797c57f1ee14c0cb3087ffd0ab2"
server_key_iv: "a1e35137f44a387098ccc5ab7a05be1c"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.230737 7456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:14.232759 7449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } attempt: 2
I20260924 13:40:14.233139 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.232998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60354)
0924 13:40:14.233060 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.233101 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 13:40:14.233510 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.233416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60364)
0924 13:40:14.233461 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.233488 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:40:14.233637 7449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:40935: Network error: Client connection negotiation failed: client connection to 127.25.240.125:40935: connect: Connection refused (error 111)
I20260924 13:40:14.258414 7456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:14.260356 7456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:14.260535 7456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:14.264652 7456 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.125:40935
I20260924 13:40:14.264743 7523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.125:40935 every 8 connection(s)
I20260924 13:40:14.265120 7456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
I20260924 13:40:14.266077 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:14.266983 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 7456
I20260924 13:40:14.267067 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-1/wal/instance
I20260924 13:40:14.267279 26561 external_mini_cluster.cc:1428] Setting key a417cdbdef5534cb6626992255d72098
I20260924 13:40:14.267899 7524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.240.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.124@KRBTEST.COM" created.
I20260924 13:40:14.277570 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.268623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.25.240.126:36877)
0924 13:40:14.268694 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.268697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.268928 (+ 231us) server_negotiation.cc:407] Connection header received
0924 13:40:14.269337 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.269339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.269353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.269436 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.270120 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.270937 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.271935 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.272310 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.272391 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.274808 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.274810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.274810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.274819 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.276208 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.276611 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.276612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.276613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.276696 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.276933 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.276934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.276934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.277042 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.277090 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.277347 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.277465 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:40:14.277733 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.268603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:36877 (local address 127.0.0.1:33376)
0924 13:40:14.268781 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.268821 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.269249 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.269489 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.269492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.269819 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.270024 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.270029 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.271045 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.271046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.271836 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.271839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.272414 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.272415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.272523 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.273088 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.273096 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.274717 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.276298 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.276305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.276306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.276527 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.276768 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.276769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.276770 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.276883 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.277108 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.277112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.277226 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.277309 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.277474 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260924 13:40:14.278318 7524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:40:14.286665 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.278602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60370)
0924 13:40:14.278656 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.278668 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.278710 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.278976 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.278978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.279133 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.279186 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.279188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.280400 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.280401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.281338 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.281340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.281875 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.281876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.281977 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.282399 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.282405 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.283673 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.285546 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.285547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.285547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.285794 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.286009 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.286009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.286009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.286078 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.286427 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.286428 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.286467 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.286529 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.286603 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:40:14.286806 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.278646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.25.240.125:40935)
0924 13:40:14.278801 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.278806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.278824 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:40:14.278853 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.278854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.278865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.278957 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.279258 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.280313 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.281419 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.281807 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.281862 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.283771 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.283785 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.283788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.283799 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.285440 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.285861 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.285864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.285864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.285921 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.286170 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.286170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.286171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.286340 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.286416 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.286563 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.286654 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260924 13:40:14.287403 7524 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:40:14.287765 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.287656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43742)
0924 13:40:14.287705 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.287747 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 13:40:14.287902 7473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
I20260924 13:40:14.288403 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.288312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43744)
0924 13:40:14.288354 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.288387 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 13:40:14.288511 7524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:41273: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab.
Entry for principal kudu/127.25.240.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab.
I20260924 13:40:14.294229 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.124:41273
--webserver_interface=127.25.240.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab
--principal=kudu/127.25.240.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:42413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:36877,127.25.240.125:40935,127.25.240.124:41273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:14.324815 7449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } attempt: 3
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:40:14.332036 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.325310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60384)
0924 13:40:14.325377 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.325387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.325450 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.325657 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.325658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.325825 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.325893 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.325895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.326573 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.326574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.327074 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.327075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.327524 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.327525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.327604 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.328091 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.328101 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.329305 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.330981 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.330982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.330982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.331163 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.331384 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.331385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.331385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.331447 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.331703 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.331704 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.331750 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.331837 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.331938 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 13:40:14.332036 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.325373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.25.240.125:40935)
0924 13:40:14.325438 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.325441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.325452 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:40:14.325518 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.325520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.325538 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.325646 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.325978 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.326486 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.327171 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.327441 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.327515 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.329393 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.329395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.329395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.329403 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.330876 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.331251 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.331252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.331253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.331312 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.331514 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.331514 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.331515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.331635 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.331711 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.331861 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.331939 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:40:14.332480 7449 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:40:14.332847 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.332756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43746)
0924 13:40:14.332798 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.332828 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 13:40:14.333163 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.333071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43748)
0924 13:40:14.333119 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.333142 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 13:40:14.333253 7449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:41273: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
I20260924 13:40:14.337695 7524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } attempt: 1
I20260924 13:40:14.338038 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.337928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43762)
0924 13:40:14.337979 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.338013 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 13:40:14.338341 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.338251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43772)
0924 13:40:14.338299 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.338322 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 13:40:14.338433 7524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:41273: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
I20260924 13:40:14.364382 7449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } attempt: 1
I20260924 13:40:14.364894 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.364710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43780)
0924 13:40:14.364798 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.364853 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 13:40:14.365310 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.365223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43796)
0924 13:40:14.365270 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.365295 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 13:40:14.365403 7449 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:41273: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
W20260924 13:40:14.375145 7531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:14.375365 7531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:14.375401 7531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:14.376506 7531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:14.376566 7531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:14.376591 7531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:14.376622 7531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:14.376649 7531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:14.378165 7531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:42413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:36877,127.25.240.125:40935,127.25.240.124:41273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab
--principal=kudu/127.25.240.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.124:41273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.240.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:14.378418 7531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:14.378638 7531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:14.383414 7536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:14.383567 7531 server_base.cc:1034] running on GCE node
W20260924 13:40:14.383432 7539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:14.383448 7537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:14.383906 7531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:14.384184 7531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:14.385336 7531 hybrid_clock.cc:648] HybridClock initialized: now 1790257214385310 us; error 40 us; skew 500 ppm
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:14.387758 7531 init.cc:377] Logged in from keytab as kudu/127.25.240.124@KRBTEST.COM (short username kudu)
I20260924 13:40:14.388236 7531 webserver.cc:460] Webserver started at http://127.25.240.124:34901/ using document root <none> and password file <none>
I20260924 13:40:14.388474 7531 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:14.388542 7531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:14.388691 7531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:14.389804 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/data/instance:
uuid: "77f47943322d4d8c9a3135284b95f858"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "9067b21a61f5255da6d70e5c449c1223"
server_key_iv: "2419a12e7980f058cc55a84e7f491629"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.390162 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/wal/instance:
uuid: "77f47943322d4d8c9a3135284b95f858"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "9067b21a61f5255da6d70e5c449c1223"
server_key_iv: "2419a12e7980f058cc55a84e7f491629"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.391464 7531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:14.392220 7546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:14.392401 7531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:14.392541 7531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/wal
uuid: "77f47943322d4d8c9a3135284b95f858"
format_stamp: "Formatted at 2026-09-24 13:40:14 on dist-test-slave-m5z1"
server_key: "9067b21a61f5255da6d70e5c449c1223"
server_key_iv: "2419a12e7980f058cc55a84e7f491629"
server_key_version: "encryptionkey@0"
I20260924 13:40:14.392625 7531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:14.402803 7531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:14.404546 7531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:14.404709 7531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:14.406570 7524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } attempt: 2
I20260924 13:40:14.406998 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.406868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43808)
0924 13:40:14.406928 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.406959 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:40:14.407335 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.407242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43812)
0924 13:40:14.407289 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.407313 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 13:40:14.407452 7524 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:41273: Network error: Client connection negotiation failed: client connection to 127.25.240.124:41273: connect: Connection refused (error 111)
I20260924 13:40:14.409044 7531 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.124:41273
I20260924 13:40:14.409091 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.124:41273 every 8 connection(s)
I20260924 13:40:14.409453 7531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
I20260924 13:40:14.410591 7599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:14.412199 7599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:14.418670 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 7531
I20260924 13:40:14.418794 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790257062855231-26561-0/minicluster-data/master-2/wal/instance
I20260924 13:40:14.418999 26561 external_mini_cluster.cc:1428] Setting key ba4d98304bdf0f778cfd24766eb63809
I20260924 13:40:14.420966 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.412902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.25.240.126:36877)
0924 13:40:14.412998 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.413001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.413153 (+ 152us) server_negotiation.cc:407] Connection header received
0924 13:40:14.413627 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.413628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.413645 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.413701 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.414264 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.414744 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.415653 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.415845 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.415895 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.417937 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.417939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.417939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.417955 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.419480 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.419997 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.419998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.419998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.420047 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.420331 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.420331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.420332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.420424 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.420478 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.420778 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.420855 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 13:40:14.420998 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.412862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:36877 (local address 127.0.0.1:33392)
0924 13:40:14.413034 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.413068 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.413489 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.413754 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.413757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.414001 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.414189 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.414193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.414819 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.414820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.415579 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.415580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.415917 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.415918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.416022 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.416523 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.416531 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.417859 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.419575 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.419578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.419580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.419918 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.420123 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.420124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.420124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.420265 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.420490 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.420497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.420652 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.420734 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.420849 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260924 13:40:14.421547 7599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:40:14.428498 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.421859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.25.240.125:40935)
0924 13:40:14.421900 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.421902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.421912 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:40:14.421939 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.421940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.421948 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.422004 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.422261 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.422745 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.423582 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.423795 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.423879 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.425669 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.425670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.425671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.425679 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.427264 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.427643 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.427644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.427644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.427701 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.427932 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.427933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.427933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.428075 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.428120 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.428334 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.428405 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:40:14.428498 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.421806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:40935 (local address 127.0.0.1:60386)
0924 13:40:14.421846 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.421856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.421891 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.422009 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.422009 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.422143 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.422184 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.422185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.422827 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.422828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.423483 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.423485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.423879 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.423880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.423953 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.424345 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.424350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.425526 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.427332 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.427333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.427333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.427567 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.427782 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.427783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.427784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.427849 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.428173 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.428173 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.428211 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.428300 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.428402 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 13:40:14.428635 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.419427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.25.240.124:41273)
0924 13:40:14.419579 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.419584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.419664 (+ 80us) server_negotiation.cc:407] Connection header received
0924 13:40:14.419729 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.419730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.419740 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.419796 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.420572 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.421752 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.422477 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.422790 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.422864 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.425028 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.425034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.425034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.425040 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.426923 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.427250 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.427253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.427254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.427328 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.427568 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.427569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.427569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.427786 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.427949 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.428335 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.428482 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260924 13:40:14.428855 7599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:14.435153 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.429074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43832)
0924 13:40:14.429125 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.429152 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.429196 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.429319 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.429319 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.429425 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.429451 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.429452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.430098 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.430099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.430602 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.430604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.430965 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.430966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.431013 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.431387 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.431391 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.432497 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.434137 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.434138 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.434138 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.434294 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.434515 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.434516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.434516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.434570 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.434815 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.434816 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.434851 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.434930 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.435031 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 13:40:14.435195 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.429167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.25.240.124:41273)
0924 13:40:14.429218 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.429220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.429240 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:40:14.429257 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.429258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.429267 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.429316 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.429532 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.430011 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.430697 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.430896 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.430965 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.432591 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.432592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.432592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.432599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.434067 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.434390 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.434391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.434391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.434439 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.434635 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.434636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.434636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.434741 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.434812 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.434950 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.435079 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":34}
I20260924 13:40:14.436286 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.429460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.25.240.126:36877)
0924 13:40:14.429497 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.429499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.429505 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:40:14.429524 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.429525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.429532 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.429629 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.430065 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.430506 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.431171 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.431373 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.431454 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.433744 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.433745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.433745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.433752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.435260 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.435638 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.435639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.435639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.435689 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.435885 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.435886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.435886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.435980 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.436102 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.436174 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.436228 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:40:14.436760 7599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858: Bootstrap starting.
I20260924 13:40:14.437685 7599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:14.437978 7599 log.cc:824] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:14.438793 7599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858: No bootstrap required, opened a new log
I20260924 13:40:14.440536 7449 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } attempt: 2
I20260924 13:40:14.441071 7599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.441224 7599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:14.441258 7599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f47943322d4d8c9a3135284b95f858, State: Initialized, Role: FOLLOWER
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:40:14.441404 7599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.441710 7605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.441838 7605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:40:14.441764 7599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 13:40:14.443727 7616 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:40:14.443806 7616 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
I20260924 13:40:14.445240 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.436942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.25.240.125:40935)
0924 13:40:14.436981 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.436982 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.437024 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:40:14.437093 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.437094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.437109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.437164 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.437643 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.438234 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.439081 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.439263 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.439317 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.441864 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.441866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.441866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.441873 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.443955 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.444323 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.444325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.444325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.444381 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.444594 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.444595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.444595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.444713 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.444863 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.445074 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.445147 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:40:14.447041 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:40:14.444245 7531 master_runner.cc:186] Error getting master registration for 127.25.240.126:36877: OK, instance_id { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" instance_seqno: 1790257214099178 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:40:14.447237 7531 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:40:14.450147 7451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.440775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43842)
0924 13:40:14.440864 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.440874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.440917 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.441042 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.441043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.441187 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.441238 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.441239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.442184 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.442185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.443026 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.443029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.443613 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.443614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.443672 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.444112 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.444117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.445692 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.448920 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.448921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.448922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.449139 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.449382 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.449382 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.449383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.449463 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.449830 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.449832 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.449881 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.449932 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.450024 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 13:40:14.450520 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.440823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.25.240.124:41273)
0924 13:40:14.440859 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.440861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.440903 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:40:14.440972 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.440972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.440982 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.441035 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.441319 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.442012 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.443132 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.443520 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.443831 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.445990 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.445991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.445991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.445998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.448842 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.449237 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.449239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.449239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.449300 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.449542 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.449542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.449543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.449737 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.450314 (+ 577us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.450354 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.450425 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:40:14.452206 7449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1: Bootstrap starting.
I20260924 13:40:14.452970 7449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:14.453310 7449 log.cc:824] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:14.454152 7449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1: No bootstrap required, opened a new log
I20260924 13:40:14.456550 7449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.456698 7449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:14.456749 7449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54db6964d3fe4a47ac828b1d9f8eb4f1, State: Initialized, Role: FOLLOWER
I20260924 13:40:14.456908 7449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.457235 7449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:14.457360 7528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.448568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.25.240.125:40935)
0924 13:40:14.448606 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.448608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.449206 (+ 598us) server_negotiation.cc:407] Connection header received
0924 13:40:14.449281 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.449282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.449291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.449321 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.450146 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.450819 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.451775 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.452052 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.452096 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.453653 (+ 1557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.453654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.453655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.453661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.456047 (+ 2386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.456435 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.456436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.456436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.456491 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.456751 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.456753 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.456753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.456880 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.456974 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.457199 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.457277 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:40:14.458159 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.448379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.25.240.126:36877)
0924 13:40:14.448582 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.448584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.449725 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 13:40:14.449829 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.449830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.449839 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.449869 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.451082 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.451795 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.452971 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.453257 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.453298 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.454906 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.454907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.454907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.454914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.456453 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.457003 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.457005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.457005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.457061 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.457373 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.457374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.457374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.457509 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.457592 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.458000 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.458077 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
W20260924 13:40:14.459383 7638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:40:14.459435 7638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:40:14.459887 7627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.459973 7627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:40:14.460004 7381 master_runner.cc:186] Error getting master registration for 127.25.240.125:40935: OK, instance_id { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" instance_seqno: 1790257214263475 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:40:14.462814 7381 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:40:14.466285 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.449252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.25.240.124:41273)
0924 13:40:14.449382 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.449385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.453786 (+ 4401us) server_negotiation.cc:407] Connection header received
0924 13:40:14.453902 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.453903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.453912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.453952 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.458807 (+ 4855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.459622 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.462038 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.462273 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.462358 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.463595 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.463596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.463597 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.463604 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.465072 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.465415 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.465416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.465417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.465479 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.465698 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.465699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.465699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.465803 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.465902 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.466020 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.466093 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260924 13:40:14.498596 7524 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } attempt: 3
I20260924 13:40:14.503698 7605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
Sep 24 13:40:14 dist-test-slave-m5z1 krb5kdc[7368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257214, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
I20260924 13:40:14.503814 7605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.504084 7605 leader_election.cc:288] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54db6964d3fe4a47ac828b1d9f8eb4f1 (127.25.240.126:36877), 28a4f2fd1da54642aa7436a3f095bca3 (127.25.240.125:40935)
I20260924 13:40:14.504539 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77f47943322d4d8c9a3135284b95f858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" is_pre_election: true
I20260924 13:40:14.504688 7423 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77f47943322d4d8c9a3135284b95f858 in term 0.
I20260924 13:40:14.505132 7550 leader_election.cc:302] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54db6964d3fe4a47ac828b1d9f8eb4f1, 77f47943322d4d8c9a3135284b95f858; no voters:
I20260924 13:40:14.505302 7605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:40:14.505364 7605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:40:14.505400 7605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:14.506319 7605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.506462 7605 leader_election.cc:288] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [CANDIDATE]: Term 1 election: Requested vote from peers 54db6964d3fe4a47ac828b1d9f8eb4f1 (127.25.240.126:36877), 28a4f2fd1da54642aa7436a3f095bca3 (127.25.240.125:40935)
I20260924 13:40:14.506821 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77f47943322d4d8c9a3135284b95f858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1"
I20260924 13:40:14.506901 7423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:14.507808 7423 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77f47943322d4d8c9a3135284b95f858 in term 1.
I20260924 13:40:14.508015 7550 leader_election.cc:302] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54db6964d3fe4a47ac828b1d9f8eb4f1, 77f47943322d4d8c9a3135284b95f858; no voters:
I20260924 13:40:14.508132 7605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:14.508311 7605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [term 1 LEADER]: Becoming Leader. State: Replica: 77f47943322d4d8c9a3135284b95f858, State: Running, Role: LEADER
I20260924 13:40:14.508419 7605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.508421 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.498922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:41273 (local address 127.0.0.1:43856)
0924 13:40:14.498962 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:14.498976 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:14.499046 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:14.499195 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:14.499196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:14.499399 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:14.499450 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.499452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.500715 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.500717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.501529 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:14.501532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.502050 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.502051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:14.502113 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.502626 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:14.502631 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:14.504184 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:14.506863 (+ 2679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.506864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.506865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.507078 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.507412 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:14.507413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:14.507414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:14.507470 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:14.507943 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:14.507945 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:14.508017 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 13:40:14.508090 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 13:40:14.508268 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 13:40:14.508817 7605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f47943322d4d8c9a3135284b95f858. Latest consensus state: current_term: 1 leader_uuid: "77f47943322d4d8c9a3135284b95f858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.508898 7605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:14.509085 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:14.498993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.25.240.124:41273)
0924 13:40:14.499063 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:14.499065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:14.499074 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:40:14.499098 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:14.499099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:14.499110 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:14.499165 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:14.499540 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.500627 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.501649 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:14.501970 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:14.502043 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:14.504280 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:14.504282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:14.504282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:14.504290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:14.506734 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.507214 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.507215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.507216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.507302 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:14.507553 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:14.507554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:14.507554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:14.507809 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:14.508694 (+ 885us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:14.508767 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 13:40:14.508939 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 13:40:14.510208 7642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:14.510445 7642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:14.510562 7524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3: Bootstrap starting.
I20260924 13:40:14.511291 7524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:14.511615 7524 log.cc:824] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:14.512475 7524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3: No bootstrap required, opened a new log
I20260924 13:40:14.513255 7423 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [term 1 FOLLOWER]: Refusing update from remote peer 77f47943322d4d8c9a3135284b95f858: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:14.513671 7640 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:14.514501 7524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.514639 7524 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:14.514685 7524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a4f2fd1da54642aa7436a3f095bca3, State: Initialized, Role: FOLLOWER
I20260924 13:40:14.514804 7524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } }
I20260924 13:40:14.515090 7524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:14.515161 7496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:14.516012 7496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [term 1 FOLLOWER]: Refusing update from remote peer 77f47943322d4d8c9a3135284b95f858: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:14.516191 7498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77f47943322d4d8c9a3135284b95f858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a4f2fd1da54642aa7436a3f095bca3" is_pre_election: true
I20260924 13:40:14.516295 7498 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77f47943322d4d8c9a3135284b95f858 in term 1.
I20260924 13:40:14.516363 7497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "77f47943322d4d8c9a3135284b95f858" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a4f2fd1da54642aa7436a3f095bca3"
I20260924 13:40:14.516595 7627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f47943322d4d8c9a3135284b95f858. Latest consensus state: current_term: 1 leader_uuid: "77f47943322d4d8c9a3135284b95f858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.516691 7627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:40:14.517735 7640 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:14.517393 7497 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77f47943322d4d8c9a3135284b95f858 in term 1.
I20260924 13:40:14.518518 7640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77f47943322d4d8c9a3135284b95f858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.518606 7640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: This master's current role is: LEADER
W20260924 13:40:14.519084 7658 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 13:40:14.519097 7627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77f47943322d4d8c9a3135284b95f858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
W20260924 13:40:14.519142 7658 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:40:14.519174 7627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:40:14.520429 7646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.520524 7646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:40:14.520830 7644 mvcc.cc:205] Tried to move back new op lower bound from 7332893550642733056 to 7332893550627123200. Current Snapshot: MvccSnapshot[applied={T|T < 7332893550642733056}]
I20260924 13:40:14.522022 7642 catalog_manager.cc:1347] Generated new cluster ID: 1649c32133fa40e8bc7fa1d29d4cdbb7
I20260924 13:40:14.522078 7642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:40:14.526330 7605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "77f47943322d4d8c9a3135284b95f858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.526420 7605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858 [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:14.526964 7661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "77f47943322d4d8c9a3135284b95f858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.527063 7661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:40:14.527158 7662 mvcc.cc:205] Tried to move back new op lower bound from 7332893550642733056 to 7332893550627123200. Current Snapshot: MvccSnapshot[applied={T|T < 7332893550642733056}]
I20260924 13:40:14.529630 7646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77f47943322d4d8c9a3135284b95f858. Latest consensus state: current_term: 1 leader_uuid: "77f47943322d4d8c9a3135284b95f858" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54db6964d3fe4a47ac828b1d9f8eb4f1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36877 } } peers { permanent_uuid: "28a4f2fd1da54642aa7436a3f095bca3" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 40935 } } peers { permanent_uuid: "77f47943322d4d8c9a3135284b95f858" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 41273 } } }
I20260924 13:40:14.529719 7646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:40:14.530077 7645 mvcc.cc:205] Tried to move back new op lower bound from 7332893550642733056 to 7332893550627123200. Current Snapshot: MvccSnapshot[applied={T|T < 7332893550642733056}]
I20260924 13:40:14.557475 7642 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:40:14.558276 7642 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:40:14.584723 7642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77f47943322d4d8c9a3135284b95f858: Generated new TSK 0
I20260924 13:40:14.584954 7642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:40:15.459860 7638 catalog_manager.cc:1259] Loaded cluster ID: 1649c32133fa40e8bc7fa1d29d4cdbb7
I20260924 13:40:15.459937 7638 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1: loading cluster ID for follower catalog manager: success
I20260924 13:40:15.460772 7638 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1: acquiring CA information for follower catalog manager: success
I20260924 13:40:15.461084 7638 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 54db6964d3fe4a47ac828b1d9f8eb4f1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:40:15.519613 7658 catalog_manager.cc:1259] Loaded cluster ID: 1649c32133fa40e8bc7fa1d29d4cdbb7
I20260924 13:40:15.519727 7658 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3: loading cluster ID for follower catalog manager: success
I20260924 13:40:15.520548 7658 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3: acquiring CA information for follower catalog manager: success
I20260924 13:40:15.520855 7658 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28a4f2fd1da54642aa7436a3f095bca3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:40:15.634420 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 7381
I20260924 13:40:15.639946 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 7456
I20260924 13:40:15.644963 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 7531
2026-09-24T13:40:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2827 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:40:15 dist-test-slave-m5z1 krb5kdc[7673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:40:15 dist-test-slave-m5z1 krb5kdc[7673](info): set up 2 sockets
Sep 24 13:40:15 dist-test-slave-m5z1 krb5kdc[7673](info): commencing operation
krb5kdc: starting...
W20260924 13:40:16.682334 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:40:16 dist-test-slave-m5z1 krb5kdc[7673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:40:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:40:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:40:16.798388 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:36727
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:46183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:36727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:16.877091 7686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:16.877306 7686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:16.877329 7686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:16.878480 7686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:16.878530 7686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:16.878543 7686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:16.878556 7686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:16.878600 7686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:16.880043 7686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:46183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:36727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:36727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:16.880281 7686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:16.880508 7686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:16.885197 7694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:16.885208 7691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:16.885197 7692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:16.885365 7686 server_base.cc:1034] running on GCE node
I20260924 13:40:16.885741 7686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:16.885974 7686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:16.887107 7686 hybrid_clock.cc:648] HybridClock initialized: now 1790257216887097 us; error 27 us; skew 500 ppm
Sep 24 13:40:16 dist-test-slave-m5z1 krb5kdc[7673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257216, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:16.889631 7686 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:16.890087 7686 webserver.cc:460] Webserver started at http://127.25.240.126:44073/ using document root <none> and password file <none>
I20260924 13:40:16.890321 7686 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:16.890365 7686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:16.890504 7686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:16.891523 7686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "5ff1e2a807d74151a4a69b42f39a6997"
format_stamp: "Formatted at 2026-09-24 13:40:16 on dist-test-slave-m5z1"
server_key: "29e97ad349737fc6bb61fe5fe8e242d8"
server_key_iv: "6735ffaf2ac3f1aea99f3a2bd0256a0a"
server_key_version: "encryptionkey@0"
I20260924 13:40:16.891872 7686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "5ff1e2a807d74151a4a69b42f39a6997"
format_stamp: "Formatted at 2026-09-24 13:40:16 on dist-test-slave-m5z1"
server_key: "29e97ad349737fc6bb61fe5fe8e242d8"
server_key_iv: "6735ffaf2ac3f1aea99f3a2bd0256a0a"
server_key_version: "encryptionkey@0"
I20260924 13:40:16.893148 7686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:40:16.893914 7701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:16.894129 7686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:40:16.894217 7686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "5ff1e2a807d74151a4a69b42f39a6997"
format_stamp: "Formatted at 2026-09-24 13:40:16 on dist-test-slave-m5z1"
server_key: "29e97ad349737fc6bb61fe5fe8e242d8"
server_key_iv: "6735ffaf2ac3f1aea99f3a2bd0256a0a"
server_key_version: "encryptionkey@0"
I20260924 13:40:16.894287 7686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:16.921968 7686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:16.923974 7686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:16.924126 7686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:16.928265 7686 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:36727
I20260924 13:40:16.928330 7753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:36727 every 8 connection(s)
I20260924 13:40:16.928696 7686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:16.929710 7754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:16.932158 7754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997: Bootstrap starting.
I20260924 13:40:16.932799 7754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:16.932888 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 7686
I20260924 13:40:16.933003 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:40:16.933169 7754 log.cc:824] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:16.933247 26561 external_mini_cluster.cc:1428] Setting key 03c350f9635955ec914bd475c2c868f2
I20260924 13:40:16.934103 7754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997: No bootstrap required, opened a new log
I20260924 13:40:16.936935 7754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff1e2a807d74151a4a69b42f39a6997" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36727 } }
I20260924 13:40:16.937121 7754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:16.937166 7754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff1e2a807d74151a4a69b42f39a6997, State: Initialized, Role: FOLLOWER
I20260924 13:40:16.937306 7754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff1e2a807d74151a4a69b42f39a6997" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36727 } }
I20260924 13:40:16.937397 7754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:40:16.937444 7754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:40:16.937500 7754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:16.938169 7754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff1e2a807d74151a4a69b42f39a6997" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36727 } }
I20260924 13:40:16.938278 7754 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ff1e2a807d74151a4a69b42f39a6997; no voters:
I20260924 13:40:16.938441 7754 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:40:16.938499 7759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:16.938701 7759 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff1e2a807d74151a4a69b42f39a6997, State: Running, Role: LEADER
I20260924 13:40:16.938709 7754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:16.938858 7759 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff1e2a807d74151a4a69b42f39a6997" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36727 } }
I20260924 13:40:16.939193 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ff1e2a807d74151a4a69b42f39a6997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff1e2a807d74151a4a69b42f39a6997" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36727 } } }
I20260924 13:40:16.939268 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:16.939839 7765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:16.940326 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff1e2a807d74151a4a69b42f39a6997. Latest consensus state: current_term: 1 leader_uuid: "5ff1e2a807d74151a4a69b42f39a6997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff1e2a807d74151a4a69b42f39a6997" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 36727 } } }
I20260924 13:40:16.940415 7765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:16.940416 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997 [sys.catalog]: This master's current role is: LEADER
Sep 24 13:40:16 dist-test-slave-m5z1 krb5kdc[7673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:16.942267 7765 catalog_manager.cc:1347] Generated new cluster ID: 5edcde1c22044f419d501f92f0b4a12e
I20260924 13:40:16.942318 7765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:40:16.945721 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:16.933757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.25.240.126:36727)
0924 13:40:16.933966 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:16.933977 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:16.934012 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:40:16.934598 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:16.934607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:16.934675 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:16.934838 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:16.935417 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:16.937027 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:16.937944 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:16.938666 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:16.938806 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:16.942066 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:16.942070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:16.942070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:16.942082 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:16.944035 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:16.944480 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:16.944483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:16.944483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:16.944581 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:16.944807 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:16.944808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:16.944808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:16.945080 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:16.945221 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:16.945370 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 13:40:16.945492 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260924 13:40:16.946761 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:40:16.958616 7765 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:40:16.959452 7765 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:40:16.971751 7765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ff1e2a807d74151a4a69b42f39a6997: Generated new TSK 0
I20260924 13:40:16.971971 7765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:40:17.000802 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 7686
2026-09-24T13:40:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1354 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:40:17 dist-test-slave-m5z1 krb5kdc[7781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:40:17 dist-test-slave-m5z1 krb5kdc[7781](info): set up 2 sockets
Sep 24 13:40:17 dist-test-slave-m5z1 krb5kdc[7781](info): commencing operation
krb5kdc: starting...
W20260924 13:40:18.037935 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:40:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:40:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:40:18.152549 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:43969
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:18.234030 7794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:18.234242 7794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:18.234278 7794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:18.235421 7794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:18.235486 7794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:18.235508 7794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:18.235536 7794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:18.235559 7794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:18.237017 7794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:43969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:18.237282 7794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:18.237515 7794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:18.242221 7800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:18.242244 7799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:18.242338 7794 server_base.cc:1034] running on GCE node
W20260924 13:40:18.242223 7802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:18.242689 7794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:18.242985 7794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:18.244131 7794 hybrid_clock.cc:648] HybridClock initialized: now 1790257218244112 us; error 41 us; skew 500 ppm
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:18.252301 7794 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:18.252730 7794 webserver.cc:460] Webserver started at http://127.25.240.126:45407/ using document root <none> and password file <none>
I20260924 13:40:18.252962 7794 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:18.253015 7794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:18.253180 7794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:18.254282 7794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "da8da141978a426ca9c1736f7a3245ae"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "a5009e57a254256ac1dff2a3273e3b65"
server_key_iv: "82adbaf496634bcd2c6911b74e9da489"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.254642 7794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "da8da141978a426ca9c1736f7a3245ae"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "a5009e57a254256ac1dff2a3273e3b65"
server_key_iv: "82adbaf496634bcd2c6911b74e9da489"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.255918 7794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:40:18.256692 7809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.256868 7794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:18.256970 7794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "da8da141978a426ca9c1736f7a3245ae"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "a5009e57a254256ac1dff2a3273e3b65"
server_key_iv: "82adbaf496634bcd2c6911b74e9da489"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.257054 7794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:18.272724 7794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:18.274891 7794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:18.275058 7794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:18.279024 7794 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:43969
I20260924 13:40:18.279062 7861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:43969 every 8 connection(s)
I20260924 13:40:18.279434 7794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:18.280457 7862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:18.282667 7862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Bootstrap starting.
I20260924 13:40:18.283175 7862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:18.283411 7862 log.cc:824] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:18.284022 7862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: No bootstrap required, opened a new log
I20260924 13:40:18.285622 7862 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:18.285785 7862 raft_consensus.cc:374] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:18.285809 7862 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8da141978a426ca9c1736f7a3245ae, State: Initialized, Role: FOLLOWER
I20260924 13:40:18.285936 7862 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:18.286015 7862 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:40:18.286038 7862 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:40:18.286067 7862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:18.286796 7862 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:18.286914 7862 leader_election.cc:302] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8da141978a426ca9c1736f7a3245ae; no voters:
I20260924 13:40:18.287084 7862 leader_election.cc:288] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:40:18.287073 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 7794
I20260924 13:40:18.287137 7865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:18.287205 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:40:18.287271 7865 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 1 LEADER]: Becoming Leader. State: Replica: da8da141978a426ca9c1736f7a3245ae, State: Running, Role: LEADER
I20260924 13:40:18.287348 7862 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:18.287415 26561 external_mini_cluster.cc:1428] Setting key 8f2ab47d887e0f40ebf5d8890d14114f
I20260924 13:40:18.287432 7865 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:18.287827 7865 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8da141978a426ca9c1736f7a3245ae. Latest consensus state: current_term: 1 leader_uuid: "da8da141978a426ca9c1736f7a3245ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } } }
I20260924 13:40:18.287904 7865 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:18.288058 7867 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da8da141978a426ca9c1736f7a3245ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } } }
I20260924 13:40:18.288173 7867 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:18.288094 7873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:18.288784 7873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:18.290489 7873 catalog_manager.cc:1347] Generated new cluster ID: f96f5d58126041b885fe31aadf8f344e
I20260924 13:40:18.290544 7873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:18.298434 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.287972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.25.240.126:43969)
0924 13:40:18.288191 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:18.288201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:18.289036 (+ 835us) server_negotiation.cc:407] Connection header received
0924 13:40:18.289154 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:18.289162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:18.289216 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:18.289360 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:18.290087 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.291154 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.291905 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.292343 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.292469 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.294790 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:18.294794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:18.294794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:18.294814 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:18.296890 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.297264 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.297266 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.297266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.297356 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.297600 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.297601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.297601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.297827 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:18.298005 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:18.298097 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 13:40:18.298237 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":80,"threads_started":1}
I20260924 13:40:18.308984 7873 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:40:18.309732 7873 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:40:18.314376 7873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Generated new TSK 0
I20260924 13:40:18.314569 7873 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:40:18.344178 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:43969
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:18.424978 7888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:18.425202 7888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:18.425225 7888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:18.426411 7888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:18.426512 7888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:18.426566 7888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:40:18.428027 7888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:18.428282 7888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:18.428565 7888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:18.433475 7894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:18.433542 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:18.433755 7888 server_base.cc:1034] running on GCE node
W20260924 13:40:18.433470 7896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:18.434021 7888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:18.434269 7888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:18.435459 7888 hybrid_clock.cc:648] HybridClock initialized: now 1790257218435409 us; error 32 us; skew 500 ppm
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:18.437911 7888 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:40:18.438374 7888 webserver.cc:460] Webserver started at http://127.25.240.65:45463/ using document root <none> and password file <none>
I20260924 13:40:18.438637 7888 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:18.438704 7888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:18.438879 7888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:18.439949 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "9556162d853848a99bb9ec68359452a4"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "89014136725891cec9baf086296e68aa"
server_key_iv: "ba99bc601aeb70001f33173a66d198b3"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.440299 7888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "9556162d853848a99bb9ec68359452a4"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "89014136725891cec9baf086296e68aa"
server_key_iv: "ba99bc601aeb70001f33173a66d198b3"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.441632 7888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:40:18.442358 7903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.442552 7888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:18.442646 7888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "9556162d853848a99bb9ec68359452a4"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "89014136725891cec9baf086296e68aa"
server_key_iv: "ba99bc601aeb70001f33173a66d198b3"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.442730 7888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:18.452993 7888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:18.454790 7888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:18.454944 7888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:18.455153 7888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:18.455449 7888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:18.455506 7888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.455546 7888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:18.455574 7888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.461732 7888 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:38447
I20260924 13:40:18.461777 8016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:38447 every 8 connection(s)
I20260924 13:40:18.462075 7888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:18.469225 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 7888
I20260924 13:40:18.469308 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:40:18.469537 26561 external_mini_cluster.cc:1428] Setting key a32b6b1c5872bbe4e390daac03444280
I20260924 13:40:18.470808 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.462720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:32781 (local address 127.25.240.126:43969)
0924 13:40:18.462803 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:18.462805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:18.462954 (+ 149us) server_negotiation.cc:407] Connection header received
0924 13:40:18.463411 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:18.463412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:18.463426 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:18.463477 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:18.464348 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.464854 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.465862 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.466057 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.466113 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.468149 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:18.468150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:18.468150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:18.468157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:18.469337 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.469839 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.469840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.469841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.469903 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.470162 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.470163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.470163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.470297 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:18.470363 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:18.470681 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 13:40:18.470746 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 13:40:18.471098 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.462679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:32781)
0924 13:40:18.462836 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:18.462870 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:18.463330 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:18.463531 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:18.463534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:18.464087 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:18.464266 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.464270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.464937 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.464938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.465791 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.465794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.466128 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.466129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.466222 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.466636 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:18.466646 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:18.468068 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:18.469425 (+ 1357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:18.469433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:18.469435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:18.469759 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:18.469998 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:18.469998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:18.469999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:18.470098 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:18.470370 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:18.470377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:18.470542 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 13:40:18.470659 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 13:40:18.470829 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":101,"threads_started":1}
I20260924 13:40:18.471590 8017 heartbeater.cc:347] Connected to a master server at 127.25.240.126:43969
I20260924 13:40:18.471707 8017 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:18.471886 8017 heartbeater.cc:511] Master 127.25.240.126:43969 requested a full tablet report, sending...
I20260924 13:40:18.472407 7826 ts_manager.cc:194] Registered new tserver with Master: 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447)
I20260924 13:40:18.473299 7826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:32781
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:40:18.494189 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:43969
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:18.574044 8022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:18.574231 8022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:18.574271 8022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:18.575419 8022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:18.575461 8022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:18.575486 8022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:40:18.576912 8022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:18.577150 8022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:18.577375 8022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:18.582238 8030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:18.582266 8028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:18.582363 8022 server_base.cc:1034] running on GCE node
W20260924 13:40:18.582289 8027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:18.582712 8022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:18.582902 8022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:18.584038 8022 hybrid_clock.cc:648] HybridClock initialized: now 1790257218584026 us; error 28 us; skew 500 ppm
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:18.586397 8022 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:40:18.586881 8022 webserver.cc:460] Webserver started at http://127.25.240.66:37269/ using document root <none> and password file <none>
I20260924 13:40:18.587141 8022 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:18.587185 8022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:18.587275 8022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:18.588340 8022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "9e8d680a8ab7454b8812fc922a88a29b"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "f089481274238d8dcc0492342a766907"
server_key_iv: "ea959b0eab8304a946e14142da3dff7d"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.588713 8022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "9e8d680a8ab7454b8812fc922a88a29b"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "f089481274238d8dcc0492342a766907"
server_key_iv: "ea959b0eab8304a946e14142da3dff7d"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.590075 8022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:40:18.590749 8037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.590930 8022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:40:18.590999 8022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "9e8d680a8ab7454b8812fc922a88a29b"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "f089481274238d8dcc0492342a766907"
server_key_iv: "ea959b0eab8304a946e14142da3dff7d"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.591050 8022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:18.612826 8022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:18.614642 8022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:18.614753 8022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:18.614923 8022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:18.615242 8022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:18.615275 8022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.615296 8022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:18.615345 8022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.621281 8022 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:46561
I20260924 13:40:18.621444 8150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:46561 every 8 connection(s)
I20260924 13:40:18.621661 8022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:18.628765 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 8022
I20260924 13:40:18.628885 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:40:18.629101 26561 external_mini_cluster.cc:1428] Setting key daa362385e09a7a7e62eb81e005c432d
I20260924 13:40:18.631065 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.622391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:37351 (local address 127.25.240.126:43969)
0924 13:40:18.622461 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:18.622463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:18.622631 (+ 168us) server_negotiation.cc:407] Connection header received
0924 13:40:18.623161 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:18.623163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:18.623175 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:18.623237 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:18.624151 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.624674 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.625637 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.625838 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.625893 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.627870 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:18.627871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:18.627871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:18.627878 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:18.629652 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.630137 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.630138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.630139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.630190 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.630464 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.630464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.630465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.630567 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:18.630617 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:18.630944 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 13:40:18.631003 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:40:18.631374 8153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.622359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:37351)
0924 13:40:18.622505 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:18.622545 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:18.623036 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:18.623296 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:18.623299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:18.623855 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:18.624040 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.624045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.624756 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.624757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.625538 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.625540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.625912 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.625913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.626013 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.626448 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:18.626456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:18.627786 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:18.629723 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:18.629729 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:18.629731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:18.630060 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:18.630256 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:18.630257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:18.630257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:18.630396 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:18.630636 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:18.630641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:18.630808 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 13:40:18.630923 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 13:40:18.631095 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260924 13:40:18.631868 8151 heartbeater.cc:347] Connected to a master server at 127.25.240.126:43969
I20260924 13:40:18.631989 8151 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:18.632184 8151 heartbeater.cc:511] Master 127.25.240.126:43969 requested a full tablet report, sending...
I20260924 13:40:18.632552 7826 ts_manager.cc:194] Registered new tserver with Master: 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561)
I20260924 13:40:18.632928 7826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:37351
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:40:18.655849 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:43969
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:18.738754 8156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:18.738951 8156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:18.738974 8156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:18.740058 8156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:18.740096 8156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:18.740123 8156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:40:18.741470 8156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:18.741720 8156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:18.741899 8156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:18.747148 8161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:18.747164 8162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:18.747164 8164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:18.747289 8156 server_base.cc:1034] running on GCE node
I20260924 13:40:18.747629 8156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:18.747906 8156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:18.749080 8156 hybrid_clock.cc:648] HybridClock initialized: now 1790257218749068 us; error 29 us; skew 500 ppm
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:18.751628 8156 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:40:18.752120 8156 webserver.cc:460] Webserver started at http://127.25.240.67:41265/ using document root <none> and password file <none>
I20260924 13:40:18.752359 8156 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:18.752425 8156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:18.752581 8156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:18.753675 8156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "2d0770d0636e41a4aff7ab3540f56ff7"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "7cc1d874ef5889ec218305c32d8e290e"
server_key_iv: "35e2f1caa7deb92c035e592e3c628257"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.754034 8156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "2d0770d0636e41a4aff7ab3540f56ff7"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "7cc1d874ef5889ec218305c32d8e290e"
server_key_iv: "35e2f1caa7deb92c035e592e3c628257"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.755378 8156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:40:18.756105 8171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.756340 8156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:18.756438 8156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "2d0770d0636e41a4aff7ab3540f56ff7"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "7cc1d874ef5889ec218305c32d8e290e"
server_key_iv: "35e2f1caa7deb92c035e592e3c628257"
server_key_version: "encryptionkey@0"
I20260924 13:40:18.756518 8156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:18.781347 8156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:18.783650 8156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:18.783813 8156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:18.784050 8156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:18.784358 8156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:18.784411 8156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.784447 8156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:18.784476 8156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:18.790908 8156 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:42935
I20260924 13:40:18.790951 8284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:42935 every 8 connection(s)
I20260924 13:40:18.791278 8156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:40:18 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257218, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:18.800503 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 8156
I20260924 13:40:18.800597 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:40:18.800748 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.791984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:53009 (local address 127.25.240.126:43969)
0924 13:40:18.792046 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:18.792048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:18.792259 (+ 211us) server_negotiation.cc:407] Connection header received
0924 13:40:18.792807 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:18.792808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:18.792820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:18.792877 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:18.793989 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.794535 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.795557 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.795755 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.795819 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.797968 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:18.797969 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:18.797969 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:18.797977 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:18.799457 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.799821 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.799822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.799822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.799872 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.800090 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.800090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.800091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.800189 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:18.800238 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:18.800552 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 13:40:18.800677 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:40:18.800783 26561 external_mini_cluster.cc:1428] Setting key 56ebf25ec572a3c60ba92fe907a40324
I20260924 13:40:18.800951 8287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.791967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:53009)
0924 13:40:18.792140 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:18.792175 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:18.792700 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:18.792971 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:18.792974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:18.793687 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:18.793881 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.793887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.794682 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.794683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.795478 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.795480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.795849 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.795850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.795960 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.796414 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:18.796422 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:18.797849 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:18.799535 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:18.799541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:18.799542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:18.799755 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:18.799929 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:18.799929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:18.799930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:18.800038 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:18.800243 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:18.800247 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:18.800410 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 13:40:18.800535 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 13:40:18.800714 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":88,"threads_started":1}
I20260924 13:40:18.801434 8285 heartbeater.cc:347] Connected to a master server at 127.25.240.126:43969
I20260924 13:40:18.801528 8285 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:18.801723 8285 heartbeater.cc:511] Master 127.25.240.126:43969 requested a full tablet report, sending...
I20260924 13:40:18.802102 7825 ts_manager.cc:194] Registered new tserver with Master: 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935)
I20260924 13:40:18.802477 7825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:53009
I20260924 13:40:18.812745 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:40:18.820493 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.814087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.25.240.126:43969)
0924 13:40:18.814166 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:18.814168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:18.814287 (+ 119us) server_negotiation.cc:407] Connection header received
0924 13:40:18.814357 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:18.814358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:18.814372 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:18.814457 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:18.815155 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.815631 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.816345 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.816541 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.816615 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.817904 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:18.817905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:18.817906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:18.817913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:18.819232 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.819686 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.819687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.819688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.819747 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:18.819940 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:18.819941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:18.819941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:18.820102 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:18.820209 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:18.820310 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 13:40:18.820390 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 13:40:18.822064 7825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:40:18.823628 7825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:40:18.829916 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.825956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:42935 (local address 127.0.0.1:50008)
0924 13:40:18.826132 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:18.826171 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:18.826232 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:18.826513 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:18.826516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:18.826518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:18.826588 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.826590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.827569 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.827571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.828704 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.828706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.829615 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.829616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.829701 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.829715 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 13:40:18.829793 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 13:40:18.829842 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":44,"threads_started":1}
I20260924 13:40:18.829954 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.826041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.25.240.67:42935)
0924 13:40:18.826254 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:18.826259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:18.826270 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:40:18.826320 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:18.826322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:18.826339 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:18.826431 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:18.826673 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.827470 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.828796 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.829474 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.829580 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.829643 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:18.829821 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 13:40:18.829869 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
I20260924 13:40:18.830348 8296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.826218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:38447 (local address 127.0.0.1:40202)
0924 13:40:18.826514 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:18.826524 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:18.826573 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:18.826801 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:18.826802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:18.826804 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:18.826845 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.826846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.827701 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.827703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.828910 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.828914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.829940 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.829941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.830020 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.830026 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:40:18.830232 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 13:40:18.830280 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":79,"threads_started":1}
I20260924 13:40:18.830379 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.826445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.25.240.65:38447)
0924 13:40:18.826642 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:18.826647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:18.826656 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:40:18.826696 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:18.826697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:18.826709 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:18.826792 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:18.826906 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.827596 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.829020 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.829763 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.829851 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.829906 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:18.830257 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 13:40:18.830299 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260924 13:40:18.831034 8219 tablet_service.cc:1467] Processing CreateTablet for tablet 3febf84b41294bf1a18613fa4dfa96df (DEFAULT_TABLE table=test-table [id=34460b340fd9410dac4a5c0f98ef312e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:18.831367 8219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3febf84b41294bf1a18613fa4dfa96df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:18.831439 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.825956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46561 (local address 127.0.0.1:43614)
0924 13:40:18.826339 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:18.826350 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:18.826407 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:18.827803 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:18.827804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:18.827806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:18.827881 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.827883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.829109 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.829110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.830200 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:18.830203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.831215 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.831216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:18.831278 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.831282 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:40:18.831332 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:40:18.831372 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":38,"thread_start_us":84,"threads_started":1}
I20260924 13:40:18.831321 7951 tablet_service.cc:1467] Processing CreateTablet for tablet 3febf84b41294bf1a18613fa4dfa96df (DEFAULT_TABLE table=test-table [id=34460b340fd9410dac4a5c0f98ef312e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:18.831671 7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3febf84b41294bf1a18613fa4dfa96df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:18.831842 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:18.826147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.25.240.66:46561)
0924 13:40:18.827623 (+ 1476us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:18.827626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:18.827635 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:40:18.827695 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:18.827697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:18.827712 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:18.827792 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:18.827959 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.829022 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.830295 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:18.831063 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:18.831128 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:18.831180 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:18.831717 (+ 537us) server_negotiation.cc:299] Negotiation successful
0924 13:40:18.831763 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1442,"thread_start_us":83,"threads_started":1}
I20260924 13:40:18.832439 8085 tablet_service.cc:1467] Processing CreateTablet for tablet 3febf84b41294bf1a18613fa4dfa96df (DEFAULT_TABLE table=test-table [id=34460b340fd9410dac4a5c0f98ef312e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:18.832760 8085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3febf84b41294bf1a18613fa4dfa96df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:18.834520 8302 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Bootstrap starting.
I20260924 13:40:18.834957 8303 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Bootstrap starting.
I20260924 13:40:18.835278 8302 tablet_bootstrap.cc:654] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:18.835623 8303 tablet_bootstrap.cc:654] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:18.835927 8302 log.cc:824] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:18.835947 8304 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Bootstrap starting.
I20260924 13:40:18.836266 8303 log.cc:824] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:18.836551 8304 tablet_bootstrap.cc:654] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:18.836826 8302 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: No bootstrap required, opened a new log
I20260924 13:40:18.836942 8302 ts_tablet_manager.cc:1397] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:40:18.836984 8303 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: No bootstrap required, opened a new log
I20260924 13:40:18.837047 8303 ts_tablet_manager.cc:1397] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:40:18.837164 8304 log.cc:824] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:18.837942 8304 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: No bootstrap required, opened a new log
I20260924 13:40:18.838012 8304 ts_tablet_manager.cc:1397] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:40:18.838701 8303 raft_consensus.cc:348] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:18.838855 8303 raft_consensus.cc:374] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:18.838893 8303 raft_consensus.cc:729] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0770d0636e41a4aff7ab3540f56ff7, State: Initialized, Role: FOLLOWER
I20260924 13:40:18.839025 8303 consensus_queue.cc:260] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:18.839072 8302 raft_consensus.cc:348] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:18.839247 8302 raft_consensus.cc:374] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:18.839279 8302 raft_consensus.cc:729] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9556162d853848a99bb9ec68359452a4, State: Initialized, Role: FOLLOWER
I20260924 13:40:18.839342 8303 ts_tablet_manager.cc:1428] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 13:40:18.839393 8302 consensus_queue.cc:260] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:18.839478 8285 heartbeater.cc:503] Master 127.25.240.126:43969 was elected leader, sending a full tablet report...
I20260924 13:40:18.839654 8302 ts_tablet_manager.cc:1428] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:40:18.839690 8017 heartbeater.cc:503] Master 127.25.240.126:43969 was elected leader, sending a full tablet report...
I20260924 13:40:18.839982 8304 raft_consensus.cc:348] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:18.840135 8304 raft_consensus.cc:374] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:18.840176 8304 raft_consensus.cc:729] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8d680a8ab7454b8812fc922a88a29b, State: Initialized, Role: FOLLOWER
I20260924 13:40:18.840287 8304 consensus_queue.cc:260] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:18.840576 8304 ts_tablet_manager.cc:1428] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:40:18.840617 8151 heartbeater.cc:503] Master 127.25.240.126:43969 was elected leader, sending a full tablet report...
W20260924 13:40:18.872304 8152 tablet.cc:2369] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:40:18.962734 8018 tablet.cc:2369] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 13:40:19.041805 8286 tablet.cc:2369] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:40:19.134408 8310 raft_consensus.cc:482] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:19.134545 8310 raft_consensus.cc:504] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.134975 8310 leader_election.cc:288] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935), 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447)
I20260924 13:40:19.138046 8311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.135208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:42935 (local address 127.25.240.66:47171)
0924 13:40:19.135397 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.135425 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.135506 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.135672 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.135673 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.135675 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:19.135758 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.135760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.136387 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.136388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.137202 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.137204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.137775 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.137777 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.137848 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.137852 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.137905 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.137936 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":16,"thread_start_us":58,"threads_started":1}
I20260924 13:40:19.138046 8298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.135318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:47171 (local address 127.25.240.67:42935)
0924 13:40:19.135371 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.135375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.135530 (+ 155us) server_negotiation.cc:407] Connection header received
0924 13:40:19.135579 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.135580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.135590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.135649 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:19.135840 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.136279 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.137310 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.137704 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.137760 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.137767 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.137930 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.137976 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:40:19.138118 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.135294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:42889 (local address 127.25.240.65:38447)
0924 13:40:19.135343 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.135345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.135439 (+ 94us) server_negotiation.cc:407] Connection header received
0924 13:40:19.135497 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.135498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.135510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.135564 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:19.135756 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.136261 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.137315 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.137727 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.137811 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.137819 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.137999 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.138039 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:40:19.138185 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.135208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:38447 (local address 127.25.240.66:42889)
0924 13:40:19.135344 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.135358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.135434 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.135583 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.135584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.135587 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:19.135672 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.135674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.136347 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.136349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.137204 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.137206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.137832 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.137833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.137917 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.137923 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.137981 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.138026 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":42,"thread_start_us":72,"threads_started":1}
I20260924 13:40:19.138641 8239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3febf84b41294bf1a18613fa4dfa96df" candidate_uuid: "9e8d680a8ab7454b8812fc922a88a29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" is_pre_election: true
I20260924 13:40:19.138705 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3febf84b41294bf1a18613fa4dfa96df" candidate_uuid: "9e8d680a8ab7454b8812fc922a88a29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9556162d853848a99bb9ec68359452a4" is_pre_election: true
I20260924 13:40:19.138809 8239 raft_consensus.cc:2454] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8d680a8ab7454b8812fc922a88a29b in term 0.
I20260924 13:40:19.138818 7971 raft_consensus.cc:2454] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8d680a8ab7454b8812fc922a88a29b in term 0.
I20260924 13:40:19.139060 8038 leader_election.cc:302] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0770d0636e41a4aff7ab3540f56ff7, 9e8d680a8ab7454b8812fc922a88a29b; no voters:
I20260924 13:40:19.139213 8310 raft_consensus.cc:2781] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:40:19.139304 8310 raft_consensus.cc:482] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:40:19.139338 8310 raft_consensus.cc:3037] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:19.140060 8310 raft_consensus.cc:504] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.140211 8310 leader_election.cc:288] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [CANDIDATE]: Term 1 election: Requested vote from peers 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935), 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447)
I20260924 13:40:19.140374 8239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3febf84b41294bf1a18613fa4dfa96df" candidate_uuid: "9e8d680a8ab7454b8812fc922a88a29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0770d0636e41a4aff7ab3540f56ff7"
I20260924 13:40:19.140394 7971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3febf84b41294bf1a18613fa4dfa96df" candidate_uuid: "9e8d680a8ab7454b8812fc922a88a29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9556162d853848a99bb9ec68359452a4"
I20260924 13:40:19.140468 8239 raft_consensus.cc:3037] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:19.140466 7971 raft_consensus.cc:3037] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:19.141183 8239 raft_consensus.cc:2454] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8d680a8ab7454b8812fc922a88a29b in term 1.
I20260924 13:40:19.141227 7971 raft_consensus.cc:2454] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8d680a8ab7454b8812fc922a88a29b in term 1.
I20260924 13:40:19.141402 8038 leader_election.cc:302] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d0770d0636e41a4aff7ab3540f56ff7, 9e8d680a8ab7454b8812fc922a88a29b; no voters:
I20260924 13:40:19.141513 8310 raft_consensus.cc:2781] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:19.141706 8310 raft_consensus.cc:686] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 1 LEADER]: Becoming Leader. State: Replica: 9e8d680a8ab7454b8812fc922a88a29b, State: Running, Role: LEADER
I20260924 13:40:19.141816 8310 consensus_queue.cc:237] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.142731 7825 catalog_manager.cc:5495] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66). New cstate: current_term: 1 leader_uuid: "9e8d680a8ab7454b8812fc922a88a29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20260924 13:40:19.181022 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.178724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.25.240.66:46561)
0924 13:40:19.178794 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.178796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.178803 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:40:19.178840 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.178841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.178861 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.178926 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:40:19.179052 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.179510 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.180334 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.180512 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.180562 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.180725 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:40:19.180829 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:40:19.180892 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.180945 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.180962 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:40:19.182909 8239 raft_consensus.cc:1261] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 1 FOLLOWER]: Refusing update from remote peer 9e8d680a8ab7454b8812fc922a88a29b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:19.182916 7971 raft_consensus.cc:1261] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Refusing update from remote peer 9e8d680a8ab7454b8812fc922a88a29b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:19.183313 8313 consensus_queue.cc:1035] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:19.183465 8310 consensus_queue.cc:1035] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:19.186714 8317 mvcc.cc:205] Tried to move back new op lower bound from 7332893569770315776 to 7332893569605591040. Current Snapshot: MvccSnapshot[applied={T|T < 7332893569770315776}]
I20260924 13:40:19.188174 7825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:19.188277 7825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:19.189014 7825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Sending DeleteTablet for 3 replicas of tablet 3febf84b41294bf1a18613fa4dfa96df
I20260924 13:40:19.189313 8085 tablet_service.cc:1514] Processing DeleteTablet for tablet 3febf84b41294bf1a18613fa4dfa96df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:19 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:43614
I20260924 13:40:19.189333 8219 tablet_service.cc:1514] Processing DeleteTablet for tablet 3febf84b41294bf1a18613fa4dfa96df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:19 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:50008
I20260924 13:40:19.189610 8326 tablet_replica.cc:330] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: stopping tablet replica
I20260924 13:40:19.189605 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 7888
I20260924 13:40:19.189724 8326 raft_consensus.cc:2229] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:19.189857 8326 raft_consensus.cc:2258] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:19.191148 8326 ts_tablet_manager.cc:1905] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:19.193058 8326 ts_tablet_manager.cc:1918] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:19.193143 8326 log.cc:1196] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/3febf84b41294bf1a18613fa4dfa96df
I20260924 13:40:19.193576 8326 ts_tablet_manager.cc:1939] T 3febf84b41294bf1a18613fa4dfa96df P 2d0770d0636e41a4aff7ab3540f56ff7: Deleting consensus metadata
I20260924 13:40:19.193672 8325 tablet_replica.cc:330] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: stopping tablet replica
I20260924 13:40:19.193813 8325 raft_consensus.cc:2229] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:40:19.193936 8325 raft_consensus.cc:2258] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:19.194193 7810 catalog_manager.cc:4841] TS 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935): tablet 3febf84b41294bf1a18613fa4dfa96df (table test-table [id=34460b340fd9410dac4a5c0f98ef312e]) successfully deleted
I20260924 13:40:19.194438 8325 ts_tablet_manager.cc:1905] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 13:40:19.195272 7812 connection.cc:535] client connection to 127.25.240.65:38447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 13:40:19.195359 7812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:40:19.195843 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.195722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:38447 (local address 127.0.0.1:40206)
0924 13:40:19.195769 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.195810 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:38447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 13:40:19.196071 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 8022
W20260924 13:40:19.196097 7812 catalog_manager.cc:4584] TS 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3febf84b41294bf1a18613fa4dfa96df: Network error: Client connection negotiation failed: client connection to 127.25.240.65:38447: connect: Connection refused (error 111)
I20260924 13:40:19.196321 8325 ts_tablet_manager.cc:1918] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:19.196386 8325 log.cc:1196] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/3febf84b41294bf1a18613fa4dfa96df
I20260924 13:40:19.201617 8300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.201481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46561 (local address 127.0.0.1:43624)
0924 13:40:19.201538 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.201583 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.66:46561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 13:40:19.201695 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 8156
I20260924 13:40:19.205936 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 7794
I20260924 13:40:19.211234 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:43969
--webserver_interface=127.25.240.126
--webserver_port=45407
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:19.298820 8328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:19.299037 8328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:19.299060 8328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:19.300143 8328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:19.300175 8328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:19.300189 8328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:19.300202 8328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:19.300215 8328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:19.301668 8328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:43969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=45407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:19.301853 8328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:19.302030 8328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:19.310395 8335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.310386 8337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.310400 8334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:19.310735 8328 server_base.cc:1034] running on GCE node
I20260924 13:40:19.310998 8328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:19.311316 8328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:19.312489 8328 hybrid_clock.cc:648] HybridClock initialized: now 1790257219312470 us; error 41 us; skew 500 ppm
Sep 24 13:40:19 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257219, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:19.315227 8328 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:19.315621 8328 webserver.cc:460] Webserver started at http://127.25.240.126:45407/ using document root <none> and password file <none>
I20260924 13:40:19.315873 8328 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:19.315932 8328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:19.317310 8328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:19.318032 8344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:19.318260 8328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:40:19.318351 8328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "da8da141978a426ca9c1736f7a3245ae"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "a5009e57a254256ac1dff2a3273e3b65"
server_key_iv: "82adbaf496634bcd2c6911b74e9da489"
server_key_version: "encryptionkey@0"
I20260924 13:40:19.318725 8328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:19.329126 8328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:19.331648 8328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:19.331766 8328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:19.335685 8328 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:43969
I20260924 13:40:19.335762 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:43969 every 8 connection(s)
I20260924 13:40:19.336160 8328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:19.337795 8397 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:40:19.338440 8397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Bootstrap starting.
I20260924 13:40:19.346246 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 8328
I20260924 13:40:19.346555 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:38447
--local_ip_for_outbound_sockets=127.25.240.65
--tserver_master_addrs=127.25.240.126:43969
--webserver_port=45463
--webserver_interface=127.25.240.65
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:19.346967 8397 log.cc:824] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:19.349885 8397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:40:19.350257 8397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Bootstrap complete.
I20260924 13:40:19.354377 8397 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:19.354813 8397 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8da141978a426ca9c1736f7a3245ae, State: Initialized, Role: FOLLOWER
I20260924 13:40:19.355002 8397 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:19.355104 8397 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:40:19.355161 8397 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:40:19.355221 8397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:40:19.356894 8397 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:19.357038 8397 leader_election.cc:302] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8da141978a426ca9c1736f7a3245ae; no voters:
I20260924 13:40:19.357242 8397 leader_election.cc:288] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:40:19.357318 8401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:40:19.357498 8401 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 2 LEADER]: Becoming Leader. State: Replica: da8da141978a426ca9c1736f7a3245ae, State: Running, Role: LEADER
I20260924 13:40:19.357543 8397 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:19.357682 8401 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:19.358068 8402 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "da8da141978a426ca9c1736f7a3245ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } } }
I20260924 13:40:19.358067 8403 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8da141978a426ca9c1736f7a3245ae. Latest consensus state: current_term: 2 leader_uuid: "da8da141978a426ca9c1736f7a3245ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } } }
I20260924 13:40:19.358168 8403 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:19.358167 8402 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:19.358789 8408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:19.359601 8408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:19.359875 8408 catalog_manager.cc:1259] Loaded cluster ID: f96f5d58126041b885fe31aadf8f344e
I20260924 13:40:19.359944 8408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:40:19.360200 8408 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 13:40:19.363400 8408 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 13:40:19.366618 8408 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:40:19.366837 8408 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Loaded TSK: 0
I20260924 13:40:19.367066 8408 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:40:19.449047 8400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:19.449292 8400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:19.449332 8400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:19.450567 8400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:19.450639 8400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:19.450682 8400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:40:19.452147 8400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:38447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=45463
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:19.452452 8400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:19.452705 8400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:19.457804 8424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.457815 8425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.457815 8427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:19.457993 8400 server_base.cc:1034] running on GCE node
I20260924 13:40:19.458228 8400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:19.458541 8400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:19.459722 8400 hybrid_clock.cc:648] HybridClock initialized: now 1790257219459695 us; error 44 us; skew 500 ppm
Sep 24 13:40:19 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257219, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:19.462543 8400 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:40:19.463028 8400 webserver.cc:460] Webserver started at http://127.25.240.65:45463/ using document root <none> and password file <none>
I20260924 13:40:19.463266 8400 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:19.463351 8400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:19.464794 8400 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:19.465552 8434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:19.465763 8400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:19.465842 8400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "9556162d853848a99bb9ec68359452a4"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "89014136725891cec9baf086296e68aa"
server_key_iv: "ba99bc601aeb70001f33173a66d198b3"
server_key_version: "encryptionkey@0"
I20260924 13:40:19.466207 8400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:19.475608 8400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:19.477937 8400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:19.478106 8400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:19.478343 8400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:19.478857 8441 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:40:19.480280 8400 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:40:19.480340 8400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:40:19.480378 8400 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:40:19.480974 8400 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:40:19.481016 8400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 13:40:19.481062 8441 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Bootstrap starting.
I20260924 13:40:19.488369 8400 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:38447
I20260924 13:40:19.488435 8548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:38447 every 8 connection(s)
I20260924 13:40:19.488793 8400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:40:19.492565 8441 log.cc:824] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:19.493134 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 8400
I20260924 13:40:19.493419 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:46561
--local_ip_for_outbound_sockets=127.25.240.66
--tserver_master_addrs=127.25.240.126:43969
--webserver_port=37269
--webserver_interface=127.25.240.66
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:19.495600 8441 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:40:19.495959 8441 tablet_bootstrap.cc:492] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Bootstrap complete.
I20260924 13:40:19.497314 8441 ts_tablet_manager.cc:1397] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260924 13:40:19.498890 8441 raft_consensus.cc:348] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.499282 8441 raft_consensus.cc:729] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9556162d853848a99bb9ec68359452a4, State: Initialized, Role: FOLLOWER
I20260924 13:40:19.499492 8441 consensus_queue.cc:260] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.499794 8441 ts_tablet_manager.cc:1428] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Sep 24 13:40:19 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257219, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:19.504168 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.489516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:33639)
0924 13:40:19.489798 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.489839 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.490277 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.493364 (+ 3087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.493369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.493816 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:19.494099 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.494107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.495801 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.495802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.497181 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.497184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.498055 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.498057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.498206 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.498841 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:19.498851 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:19.500520 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:19.502545 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:19.502552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:19.502554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:19.502783 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:19.503056 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:19.503057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:19.503058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:19.503169 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:19.503540 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:19.503546 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:19.503684 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.503793 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.503968 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
I20260924 13:40:19.504184 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.489630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:33639 (local address 127.25.240.126:43969)
0924 13:40:19.492861 (+ 3231us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.492871 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.492918 (+ 47us) server_negotiation.cc:407] Connection header received
0924 13:40:19.493068 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.493086 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.493149 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.493330 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:19.494245 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.495662 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.497301 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.497928 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.498516 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.500664 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:19.500668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:19.500669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:19.500680 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:19.502436 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:19.502886 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:19.502888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:19.502889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:19.502983 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:19.503239 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:19.503240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:19.503240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:19.503460 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:19.503547 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.503846 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.503968 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3158,"thread_start_us":81,"threads_started":1}
I20260924 13:40:19.504866 8549 heartbeater.cc:347] Connected to a master server at 127.25.240.126:43969
I20260924 13:40:19.504982 8549 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:19.505162 8549 heartbeater.cc:511] Master 127.25.240.126:43969 requested a full tablet report, sending...
I20260924 13:40:19.505889 8361 ts_manager.cc:194] Registered new tserver with Master: 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447)
I20260924 13:40:19.507032 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:33639
I20260924 13:40:19.507869 8549 heartbeater.cc:503] Master 127.25.240.126:43969 was elected leader, sending a full tablet report...
I20260924 13:40:19.512789 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.506467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:38447 (local address 127.0.0.1:40222)
0924 13:40:19.509706 (+ 3239us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.509738 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.509793 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.510034 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.510037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.510039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:19.510130 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.510132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.510943 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.510945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.511783 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.511786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.512437 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.512438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.512538 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.512551 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.512671 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.512732 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3206,"thread_start_us":99,"threads_started":1}
I20260924 13:40:19.513048 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.506712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.25.240.65:38447)
0924 13:40:19.509896 (+ 3184us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.509899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.509906 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:40:19.509931 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.509933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.509943 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.510029 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:19.510222 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.510857 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.511879 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.512341 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.512426 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.512485 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.512920 (+ 435us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.512971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3157,"thread_start_us":65,"threads_started":1}
I20260924 13:40:19.513477 8483 tablet_service.cc:1514] Processing DeleteTablet for tablet 3febf84b41294bf1a18613fa4dfa96df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:19 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:40222
I20260924 13:40:19.514099 8558 tablet_replica.cc:330] stopping tablet replica
I20260924 13:40:19.514222 8558 raft_consensus.cc:2229] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:19.514317 8558 raft_consensus.cc:2258] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:19.514683 8558 ts_tablet_manager.cc:1905] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:19.516713 8558 ts_tablet_manager.cc:1918] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:19.516799 8558 log.cc:1196] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/3febf84b41294bf1a18613fa4dfa96df
I20260924 13:40:19.517068 8558 ts_tablet_manager.cc:1939] T 3febf84b41294bf1a18613fa4dfa96df P 9556162d853848a99bb9ec68359452a4: Deleting consensus metadata
I20260924 13:40:19.517584 8347 catalog_manager.cc:4841] TS 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447): tablet 3febf84b41294bf1a18613fa4dfa96df (table test-table [id=34460b340fd9410dac4a5c0f98ef312e]) successfully deleted
W20260924 13:40:19.584367 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:19.584560 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:19.584582 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:19.585767 8554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:19.585807 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:19.585834 8554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:40:19.587275 8554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:46561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=37269
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:19.587503 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:19.587723 8554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:19.592653 8566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.592641 8563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:19.592778 8554 server_base.cc:1034] running on GCE node
W20260924 13:40:19.592700 8564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:19.593089 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:19.593366 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:19.594529 8554 hybrid_clock.cc:648] HybridClock initialized: now 1790257219594509 us; error 46 us; skew 500 ppm
Sep 24 13:40:19 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257219, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:19.597005 8554 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:40:19.597458 8554 webserver.cc:460] Webserver started at http://127.25.240.66:37269/ using document root <none> and password file <none>
I20260924 13:40:19.597720 8554 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:19.597805 8554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:19.599047 8554 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:19.599694 8573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:19.599857 8554 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:19.599956 8554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "9e8d680a8ab7454b8812fc922a88a29b"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "f089481274238d8dcc0492342a766907"
server_key_iv: "ea959b0eab8304a946e14142da3dff7d"
server_key_version: "encryptionkey@0"
I20260924 13:40:19.600289 8554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:19.623000 8554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:19.625499 8554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:19.625679 8554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:19.625922 8554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:19.626369 8580 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 13:40:19.627684 8580 ts_tablet_manager.cc:1852] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 13:40:19.627952 8580 ts_tablet_manager.cc:1905] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:19.630017 8580 ts_tablet_manager.cc:1918] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:19.630090 8580 log.cc:1196] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/3febf84b41294bf1a18613fa4dfa96df
I20260924 13:40:19.630348 8580 ts_tablet_manager.cc:1939] T 3febf84b41294bf1a18613fa4dfa96df P 9e8d680a8ab7454b8812fc922a88a29b: Deleting consensus metadata
I20260924 13:40:19.630551 8554 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 13:40:19.630625 8554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 13:40:19.630673 8554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:19.630703 8554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:19.636474 8554 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:46561
I20260924 13:40:19.636560 8687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:46561 every 8 connection(s)
I20260924 13:40:19.636861 8554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:40:19.639781 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 8554
I20260924 13:40:19.640053 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:42935
--local_ip_for_outbound_sockets=127.25.240.67
--tserver_master_addrs=127.25.240.126:43969
--webserver_port=41265
--webserver_interface=127.25.240.67
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 13:40:19 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257219, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:19.646596 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.637503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:40481)
0924 13:40:19.637699 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.637739 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.638195 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.638422 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.638427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.638666 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:19.638845 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.638849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.639512 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.639513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.640840 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.640844 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.641229 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.641230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.641385 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.641997 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:19.642005 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:19.643429 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:19.645298 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:19.645303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:19.645305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:19.645512 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:19.645764 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:19.645764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:19.645765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:19.645861 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:19.646158 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:19.646162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:19.646266 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.646337 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.646449 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":124,"threads_started":1}
I20260924 13:40:19.646644 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.637588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:40481 (local address 127.25.240.126:43969)
0924 13:40:19.637647 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.637649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.637831 (+ 182us) server_negotiation.cc:407] Connection header received
0924 13:40:19.638280 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.638281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.638292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.638354 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:19.638920 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.639435 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.640934 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.641147 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.641204 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.643539 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:19.643541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:19.643542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:19.643550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:19.645215 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:19.645629 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:19.645630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:19.645630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:19.645697 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:19.645939 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:19.645940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:19.645940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:19.646090 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:19.646166 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.646403 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.646511 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:40:19.647042 8688 heartbeater.cc:347] Connected to a master server at 127.25.240.126:43969
I20260924 13:40:19.647150 8688 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:19.647351 8688 heartbeater.cc:511] Master 127.25.240.126:43969 requested a full tablet report, sending...
I20260924 13:40:19.647735 8361 ts_manager.cc:194] Registered new tserver with Master: 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561)
I20260924 13:40:19.648108 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:40481
W20260924 13:40:19.722338 8691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:19.722529 8691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:19.722553 8691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:19.723730 8691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:19.723771 8691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:19.723824 8691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:40:19.725257 8691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:42935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=41265
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:19.725498 8691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:19.725751 8691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:19.730609 8696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.730635 8697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.730635 8699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:19.730847 8691 server_base.cc:1034] running on GCE node
I20260924 13:40:19.731065 8691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:19.731304 8691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:19.732466 8691 hybrid_clock.cc:648] HybridClock initialized: now 1790257219732444 us; error 49 us; skew 500 ppm
Sep 24 13:40:19 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257219, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:19.734993 8691 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:40:19.735410 8691 webserver.cc:460] Webserver started at http://127.25.240.67:41265/ using document root <none> and password file <none>
I20260924 13:40:19.735685 8691 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:19.735769 8691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:19.737025 8691 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:19.737604 8706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:19.737771 8691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:19.737867 8691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "2d0770d0636e41a4aff7ab3540f56ff7"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "7cc1d874ef5889ec218305c32d8e290e"
server_key_iv: "35e2f1caa7deb92c035e592e3c628257"
server_key_version: "encryptionkey@0"
I20260924 13:40:19.738204 8691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:19.750152 8691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:19.751839 8691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:19.751991 8691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:19.752202 8691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:19.752516 8691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:19.752570 8691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:19.752612 8691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:19.752655 8691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:19.758538 8691 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:42935
I20260924 13:40:19.758575 8819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:42935 every 8 connection(s)
I20260924 13:40:19.758960 8691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
Sep 24 13:40:19 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257219, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:19.765046 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 8691
I20260924 13:40:19.768325 8552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.759677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:50273 (local address 127.25.240.126:43969)
0924 13:40:19.759743 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.759745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.759952 (+ 207us) server_negotiation.cc:407] Connection header received
0924 13:40:19.760424 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.760425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.760435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.760486 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:19.761345 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.761836 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.762793 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.762971 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.763022 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.765244 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:19.765246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:19.765247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:19.765254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:19.766864 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:19.767329 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:19.767330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:19.767331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:19.767388 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:19.767682 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:19.767683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:19.767683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:19.767789 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:19.767846 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.768195 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.768261 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 13:40:19.768592 8822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.759683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:50273)
0924 13:40:19.759827 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.759859 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.760339 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.760538 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.760541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.761089 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:19.761272 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.761277 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.761914 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.761915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.762720 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.762722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.763046 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.763046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.763147 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.763716 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:19.763724 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:19.765077 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:19.766945 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:19.766952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:19.766955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:19.767251 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:19.767465 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:19.767466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:19.767467 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:19.767599 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:19.767882 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:19.767889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:19.768054 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.768177 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.768327 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260924 13:40:19.769169 8820 heartbeater.cc:347] Connected to a master server at 127.25.240.126:43969
I20260924 13:40:19.769258 8820 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:19.769394 8820 heartbeater.cc:511] Master 127.25.240.126:43969 requested a full tablet report, sending...
I20260924 13:40:19.769804 8361 ts_manager.cc:194] Registered new tserver with Master: 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935)
I20260924 13:40:19.770161 8361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:50273
I20260924 13:40:19.772550 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.765429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.25.240.126:43969)
0924 13:40:19.765538 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.765540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.765686 (+ 146us) server_negotiation.cc:407] Connection header received
0924 13:40:19.765710 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.765711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.765723 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.765784 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:19.766556 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.767025 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.768531 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.768783 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.768844 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.770291 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:19.770293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:19.770293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:19.770300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:19.771599 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:19.771938 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:19.771939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:19.771939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:19.771988 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:19.772122 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:19.772123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:19.772123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:19.772222 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:19.772332 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.772435 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.772493 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260924 13:40:19.773219 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:40:19.775559 8824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.773528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47090 (local address 127.25.240.126:43969)
0924 13:40:19.773595 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.773597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.773658 (+ 61us) server_negotiation.cc:407] Connection header received
0924 13:40:19.773718 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.773718 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.773745 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.773804 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:40:19.773873 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.774302 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.774986 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.775165 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.775214 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.775336 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:40:19.775419 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:40:19.775472 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.775501 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.775516 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 13:40:19.775841 8361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:40:19.777501 8361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:40:19.779627 8483 tablet_service.cc:1467] Processing CreateTablet for tablet ef05f011cbb14d909b63d4a3143c1101 (DEFAULT_TABLE table=test-table [id=19b8b9355ad349bebb0c7db3b8e49361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:19.779822 8483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef05f011cbb14d909b63d4a3143c1101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:19.781183 8441 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Bootstrap starting.
I20260924 13:40:19.781491 8441 tablet_bootstrap.cc:654] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:19.782006 8441 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: No bootstrap required, opened a new log
I20260924 13:40:19.782068 8441 ts_tablet_manager.cc:1397] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:19.782176 8441 raft_consensus.cc:348] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.782272 8441 raft_consensus.cc:374] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:19.782320 8441 raft_consensus.cc:729] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9556162d853848a99bb9ec68359452a4, State: Initialized, Role: FOLLOWER
I20260924 13:40:19.782366 8441 consensus_queue.cc:260] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.782459 8441 ts_tablet_manager.cc:1428] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:19.783555 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.779602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50012 (local address 127.25.240.67:42935)
0924 13:40:19.779757 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.779761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.779770 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:40:19.779801 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.779803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.779816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.779895 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:19.780047 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.781078 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.782490 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.783206 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.783283 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.783343 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.783438 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.783480 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260924 13:40:19.783572 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.779330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46561 (local address 127.0.0.1:43630)
0924 13:40:19.779378 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.779388 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.779424 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.779806 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.779807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.779808 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:19.779849 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.779851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.781300 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.781301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.782385 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.782387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.783367 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.783368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.783428 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.783430 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.783485 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.783515 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:40:19.783617 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.779425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.25.240.66:46561)
0924 13:40:19.779617 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.779623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.779633 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:40:19.779681 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.779683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.779701 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.779925 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:19.779986 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.781198 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.782490 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.783298 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.783363 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.783412 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.783497 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.783530 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":80,"threads_started":1}
I20260924 13:40:19.783571 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.779387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:42935 (local address 127.0.0.1:50012)
0924 13:40:19.779535 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.779544 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.779585 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.779906 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.779907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.779909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:19.779962 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.779964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.781197 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.781199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.782386 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.782387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.783286 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.783286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.783354 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.783359 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.783423 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.783490 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260924 13:40:19.784148 8622 tablet_service.cc:1467] Processing CreateTablet for tablet ef05f011cbb14d909b63d4a3143c1101 (DEFAULT_TABLE table=test-table [id=19b8b9355ad349bebb0c7db3b8e49361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:19.784271 8754 tablet_service.cc:1467] Processing CreateTablet for tablet ef05f011cbb14d909b63d4a3143c1101 (DEFAULT_TABLE table=test-table [id=19b8b9355ad349bebb0c7db3b8e49361]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:19.784473 8622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef05f011cbb14d909b63d4a3143c1101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:19.784549 8754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef05f011cbb14d909b63d4a3143c1101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:19.786659 8580 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Bootstrap starting.
I20260924 13:40:19.787346 8580 tablet_bootstrap.cc:654] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:19.787638 8580 log.cc:824] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:19.787667 8830 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: Bootstrap starting.
I20260924 13:40:19.788326 8830 tablet_bootstrap.cc:654] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:19.788478 8580 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: No bootstrap required, opened a new log
I20260924 13:40:19.788543 8580 ts_tablet_manager.cc:1397] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:40:19.788858 8830 log.cc:824] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:19.789510 8830 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: No bootstrap required, opened a new log
I20260924 13:40:19.789608 8830 ts_tablet_manager.cc:1397] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:40:19.790123 8580 raft_consensus.cc:348] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.790271 8580 raft_consensus.cc:374] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:19.790315 8580 raft_consensus.cc:729] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e8d680a8ab7454b8812fc922a88a29b, State: Initialized, Role: FOLLOWER
I20260924 13:40:19.790437 8580 consensus_queue.cc:260] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.790717 8580 ts_tablet_manager.cc:1428] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:40:19.790741 8688 heartbeater.cc:503] Master 127.25.240.126:43969 was elected leader, sending a full tablet report...
I20260924 13:40:19.791383 8830 raft_consensus.cc:348] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.791579 8830 raft_consensus.cc:374] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:19.791617 8830 raft_consensus.cc:729] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d0770d0636e41a4aff7ab3540f56ff7, State: Initialized, Role: FOLLOWER
I20260924 13:40:19.791723 8830 consensus_queue.cc:260] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.791971 8830 ts_tablet_manager.cc:1428] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:40:19.792028 8820 heartbeater.cc:503] Master 127.25.240.126:43969 was elected leader, sending a full tablet report...
I20260924 13:40:19.851419 8833 raft_consensus.cc:482] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:19.851553 8833 raft_consensus.cc:504] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.851907 8833 leader_election.cc:288] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935), 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447)
I20260924 13:40:19.854872 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.852132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:38447 (local address 127.25.240.66:57985)
0924 13:40:19.852188 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.852201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.852249 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.852453 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.852454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.852456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:19.852550 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.852552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.853216 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.853217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.854037 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.854039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.854609 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.854610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.854693 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.854696 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.854763 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.854800 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 13:40:19.854872 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.852186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:57985 (local address 127.25.240.65:38447)
0924 13:40:19.852231 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.852233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.852243 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:40:19.852341 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.852342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.852353 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.852403 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:19.852634 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.853130 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.854131 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.854526 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.854613 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.854623 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.854769 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.854809 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:40:19.855069 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.852188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:32905 (local address 127.25.240.67:42935)
0924 13:40:19.852300 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.852302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.852415 (+ 113us) server_negotiation.cc:407] Connection header received
0924 13:40:19.852522 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.852523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.852534 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.852596 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:19.852830 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.853287 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.854203 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.854683 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.854776 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.854785 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.854924 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.854983 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 13:40:19.855242 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.852150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:42935 (local address 127.25.240.66:32905)
0924 13:40:19.852360 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.852370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:19.852436 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:19.852690 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:19.852691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:19.852692 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:19.852748 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.852749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.853372 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.853373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.854113 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:19.854115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.854777 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.854778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:19.854853 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.854856 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:40:19.855051 (+ 195us) client_negotiation.cc:240] Negotiation successful
0924 13:40:19.855091 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":119,"threads_started":1}
I20260924 13:40:19.855501 8503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef05f011cbb14d909b63d4a3143c1101" candidate_uuid: "9e8d680a8ab7454b8812fc922a88a29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9556162d853848a99bb9ec68359452a4" is_pre_election: true
I20260924 13:40:19.855664 8503 raft_consensus.cc:2454] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8d680a8ab7454b8812fc922a88a29b in term 0.
I20260924 13:40:19.855621 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef05f011cbb14d909b63d4a3143c1101" candidate_uuid: "9e8d680a8ab7454b8812fc922a88a29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" is_pre_election: true
I20260924 13:40:19.855796 8774 raft_consensus.cc:2454] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e8d680a8ab7454b8812fc922a88a29b in term 0.
I20260924 13:40:19.855924 8576 leader_election.cc:302] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9556162d853848a99bb9ec68359452a4, 9e8d680a8ab7454b8812fc922a88a29b; no voters:
I20260924 13:40:19.856058 8833 raft_consensus.cc:2781] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:40:19.856144 8833 raft_consensus.cc:482] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:40:19.856173 8833 raft_consensus.cc:3037] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:19.856842 8833 raft_consensus.cc:504] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.856989 8833 leader_election.cc:288] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [CANDIDATE]: Term 1 election: Requested vote from peers 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935), 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447)
I20260924 13:40:19.857187 8503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef05f011cbb14d909b63d4a3143c1101" candidate_uuid: "9e8d680a8ab7454b8812fc922a88a29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9556162d853848a99bb9ec68359452a4"
I20260924 13:40:19.857225 8774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef05f011cbb14d909b63d4a3143c1101" candidate_uuid: "9e8d680a8ab7454b8812fc922a88a29b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d0770d0636e41a4aff7ab3540f56ff7"
I20260924 13:40:19.857275 8503 raft_consensus.cc:3037] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:19.857308 8774 raft_consensus.cc:3037] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:19.858170 8503 raft_consensus.cc:2454] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8d680a8ab7454b8812fc922a88a29b in term 1.
I20260924 13:40:19.858251 8774 raft_consensus.cc:2454] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e8d680a8ab7454b8812fc922a88a29b in term 1.
I20260924 13:40:19.858392 8576 leader_election.cc:302] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9556162d853848a99bb9ec68359452a4, 9e8d680a8ab7454b8812fc922a88a29b; no voters:
I20260924 13:40:19.858500 8833 raft_consensus.cc:2781] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:19.858668 8833 raft_consensus.cc:686] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 1 LEADER]: Becoming Leader. State: Replica: 9e8d680a8ab7454b8812fc922a88a29b, State: Running, Role: LEADER
I20260924 13:40:19.858767 8833 consensus_queue.cc:237] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:19.859547 8361 catalog_manager.cc:5495] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66). New cstate: current_term: 1 leader_uuid: "9e8d680a8ab7454b8812fc922a88a29b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } health_report { overall_health: UNKNOWN } } }
I20260924 13:40:19.875174 8827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.872776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.25.240.66:46561)
0924 13:40:19.872820 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:19.872822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:19.872831 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:40:19.872912 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:19.872912 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:19.872938 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:19.873013 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:40:19.873115 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.873632 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.874364 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:19.874564 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:19.874638 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:19.874807 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:40:19.874909 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:40:19.874995 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:19.875036 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 13:40:19.875056 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:40:19.876940 8774 raft_consensus.cc:1261] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 1 FOLLOWER]: Refusing update from remote peer 9e8d680a8ab7454b8812fc922a88a29b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:19.876940 8503 raft_consensus.cc:1261] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Refusing update from remote peer 9e8d680a8ab7454b8812fc922a88a29b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:19.877339 8833 consensus_queue.cc:1035] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:19.877466 8836 consensus_queue.cc:1035] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:19.880271 8839 mvcc.cc:205] Tried to move back new op lower bound from 7332893572613107712 to 7332893572542099456. Current Snapshot: MvccSnapshot[applied={T|T < 7332893572613107712}]
I20260924 13:40:19.881049 8841 mvcc.cc:205] Tried to move back new op lower bound from 7332893572613107712 to 7332893572542099456. Current Snapshot: MvccSnapshot[applied={T|T < 7332893572613107712}]
I20260924 13:40:19.882089 8361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:19.882187 8361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:19.882942 8361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Sending DeleteTablet for 3 replicas of tablet ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:19.883220 8622 tablet_service.cc:1514] Processing DeleteTablet for tablet ef05f011cbb14d909b63d4a3143c1101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:19 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:43630
I20260924 13:40:19.883338 8483 tablet_service.cc:1514] Processing DeleteTablet for tablet ef05f011cbb14d909b63d4a3143c1101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:19 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:40222
I20260924 13:40:19.883337 8754 tablet_service.cc:1514] Processing DeleteTablet for tablet ef05f011cbb14d909b63d4a3143c1101 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:19 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:50012
I20260924 13:40:19.883422 8558 tablet_replica.cc:330] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: stopping tablet replica
I20260924 13:40:19.883469 8558 raft_consensus.cc:2229] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:19.883538 8558 raft_consensus.cc:2258] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:19.883788 8848 tablet_replica.cc:330] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: stopping tablet replica
I20260924 13:40:19.883797 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 8400
I20260924 13:40:19.883826 8558 ts_tablet_manager.cc:1905] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:19.884529 8848 raft_consensus.cc:2229] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:19.884613 8848 raft_consensus.cc:2258] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:19.884974 8848 ts_tablet_manager.cc:1905] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:19.886492 8848 ts_tablet_manager.cc:1918] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:19.886540 8848 log.cc:1196] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:19.886737 8848 ts_tablet_manager.cc:1939] T ef05f011cbb14d909b63d4a3143c1101 P 2d0770d0636e41a4aff7ab3540f56ff7: Deleting consensus metadata
I20260924 13:40:19.887145 8345 catalog_manager.cc:4841] TS 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935): tablet ef05f011cbb14d909b63d4a3143c1101 (table test-table [id=19b8b9355ad349bebb0c7db3b8e49361]) successfully deleted
I20260924 13:40:19.889690 8847 tablet_replica.cc:330] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: stopping tablet replica
I20260924 13:40:19.889804 8847 raft_consensus.cc:2229] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:40:19.889915 8847 raft_consensus.cc:2258] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:19.890300 8847 ts_tablet_manager.cc:1905] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 13:40:19.891047 8347 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:38447 (error 108)
I20260924 13:40:19.891309 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 8554
I20260924 13:40:19.891631 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.891521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:38447 (local address 127.0.0.1:40228)
0924 13:40:19.891568 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.891599 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:38447: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:40:19.891992 8347 catalog_manager.cc:4584] TS 9556162d853848a99bb9ec68359452a4 (127.25.240.65:38447): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef05f011cbb14d909b63d4a3143c1101: Network error: Client connection negotiation failed: client connection to 127.25.240.65:38447: connect: Connection refused (error 111)
I20260924 13:40:19.896215 8826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:19.896090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46561 (local address 127.0.0.1:43646)
0924 13:40:19.896147 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:19.896177 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.66:46561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:40:19.896338 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 8691
I20260924 13:40:19.901010 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 8328
I20260924 13:40:19.906387 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:43969
--webserver_interface=127.25.240.126
--webserver_port=45407
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:43969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:19.985881 8850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:19.986084 8850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:19.986109 8850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:19.987237 8850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:19.987277 8850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:19.987298 8850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:19.987351 8850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:19.987370 8850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:19.988803 8850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:43969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:43969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=45407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:19.989053 8850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:19.989281 8850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:19.993860 8858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.993883 8856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:19.993849 8855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:19.994102 8850 server_base.cc:1034] running on GCE node
I20260924 13:40:19.994361 8850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:19.994694 8850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:19.995841 8850 hybrid_clock.cc:648] HybridClock initialized: now 1790257219995828 us; error 28 us; skew 500 ppm
Sep 24 13:40:19 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257219, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:19.998266 8850 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:19.998751 8850 webserver.cc:460] Webserver started at http://127.25.240.126:45407/ using document root <none> and password file <none>
I20260924 13:40:19.998976 8850 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:19.999046 8850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:20.000319 8850 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:20.000985 8865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:20.001153 8850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:20.001253 8850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "da8da141978a426ca9c1736f7a3245ae"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "a5009e57a254256ac1dff2a3273e3b65"
server_key_iv: "82adbaf496634bcd2c6911b74e9da489"
server_key_version: "encryptionkey@0"
I20260924 13:40:20.001650 8850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:20.022068 8850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:20.024356 8850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:20.024524 8850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:20.028445 8850 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:43969
I20260924 13:40:20.028479 8917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:43969 every 8 connection(s)
I20260924 13:40:20.028862 8850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:20.030453 8918 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:40:20.030836 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 8850
I20260924 13:40:20.031155 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:38447
--local_ip_for_outbound_sockets=127.25.240.65
--tserver_master_addrs=127.25.240.126:43969
--webserver_port=45463
--webserver_interface=127.25.240.65
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:20.031400 8918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Bootstrap starting.
I20260924 13:40:20.039906 8918 log.cc:824] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:20.042276 8918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:40:20.042578 8918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: Bootstrap complete.
I20260924 13:40:20.044859 8918 raft_consensus.cc:348] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:20.045078 8918 raft_consensus.cc:729] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: da8da141978a426ca9c1736f7a3245ae, State: Initialized, Role: FOLLOWER
I20260924 13:40:20.045251 8918 consensus_queue.cc:260] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:20.045308 8918 raft_consensus.cc:388] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:40:20.045372 8918 raft_consensus.cc:482] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:40:20.045415 8918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 2 FOLLOWER]: Advancing to term 3
I20260924 13:40:20.046438 8918 raft_consensus.cc:504] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:20.046521 8918 leader_election.cc:302] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da8da141978a426ca9c1736f7a3245ae; no voters:
I20260924 13:40:20.046722 8918 leader_election.cc:288] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 13:40:20.046761 8922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 3 FOLLOWER]: Leader election won for term 3
I20260924 13:40:20.046918 8922 raft_consensus.cc:686] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [term 3 LEADER]: Becoming Leader. State: Replica: da8da141978a426ca9c1736f7a3245ae, State: Running, Role: LEADER
I20260924 13:40:20.046984 8918 sys_catalog.cc:564] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:20.047096 8922 consensus_queue.cc:237] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } }
I20260924 13:40:20.047447 8922 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader da8da141978a426ca9c1736f7a3245ae. Latest consensus state: current_term: 3 leader_uuid: "da8da141978a426ca9c1736f7a3245ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } } }
I20260924 13:40:20.047448 8924 sys_catalog.cc:455] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "da8da141978a426ca9c1736f7a3245ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da8da141978a426ca9c1736f7a3245ae" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 43969 } } }
I20260924 13:40:20.047724 8924 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:20.047638 8922 sys_catalog.cc:458] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:20.048867 8938 catalog_manager.cc:1259] Loaded cluster ID: f96f5d58126041b885fe31aadf8f344e
I20260924 13:40:20.048918 8938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: loading cluster ID for follower catalog manager: success
W20260924 13:40:20.049130 8938 catalog_manager.cc:1576] T 00000000000000000000000000000000 P da8da141978a426ca9c1736f7a3245ae: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260924 13:40:20.049175 8938 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260924 13:40:20.049214 8931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:20.049549 8931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:20.049685 8931 catalog_manager.cc:1259] Loaded cluster ID: f96f5d58126041b885fe31aadf8f344e
I20260924 13:40:20.049716 8931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 13:40:20.049849 8931 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f63dd7099ad google::LogMessage::Fail() at ??:0
@ 0x7f63dd70c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f63dd7094de google::LogMessage::Flush() at ??:0
@ 0x7f63dd70aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f63e562a0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f63e56338b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f63df5d70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f63df5cc954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f63e04306db start_thread at ??:0
@ 0x7f63dc20761f clone at ??:0
W20260924 13:40:20.123262 8920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:20.123445 8920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:20.123467 8920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:20.124577 8920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:20.124615 8920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:20.124641 8920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:40:20.126142 8920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:38447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=45463
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:20.126377 8920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:20.126621 8920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:20.131618 8954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:20.131637 8957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:20.131832 8920 server_base.cc:1034] running on GCE node
W20260924 13:40:20.131637 8955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:20.132138 8920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:20.132409 8920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:20.133597 8920 hybrid_clock.cc:648] HybridClock initialized: now 1790257220133574 us; error 39 us; skew 500 ppm
Sep 24 13:40:20 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257220, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:20.136138 8920 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:40:20.136569 8920 webserver.cc:460] Webserver started at http://127.25.240.65:45463/ using document root <none> and password file <none>
I20260924 13:40:20.136826 8920 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:20.136895 8920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:20.138298 8920 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:20.139073 8964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:20.139254 8920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:20.139350 8920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "9556162d853848a99bb9ec68359452a4"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "89014136725891cec9baf086296e68aa"
server_key_iv: "ba99bc601aeb70001f33173a66d198b3"
server_key_version: "encryptionkey@0"
I20260924 13:40:20.139701 8920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:20.152212 8920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:20.154379 8920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:20.154551 8920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:20.154774 8920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:20.155202 8971 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:40:20.157086 8920 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:40:20.157143 8920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:40:20.157191 8920 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:40:20.157914 8920 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:40:20.157968 8920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:20.158051 8971 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Bootstrap starting.
I20260924 13:40:20.159955 8971 log.cc:824] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:20.163087 8971 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:40:20.163389 8971 tablet_bootstrap.cc:492] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Bootstrap complete.
I20260924 13:40:20.163544 8971 ts_tablet_manager.cc:1397] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 13:40:20.165071 8920 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:38447
I20260924 13:40:20.165103 9079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:38447 every 8 connection(s)
I20260924 13:40:20.165444 8920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:40:20.165336 8971 raft_consensus.cc:348] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:20.165747 8971 raft_consensus.cc:729] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9556162d853848a99bb9ec68359452a4, State: Initialized, Role: FOLLOWER
I20260924 13:40:20.165889 8971 consensus_queue.cc:260] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:20.166157 8971 ts_tablet_manager.cc:1428] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 13:40:20.166378 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 8920
I20260924 13:40:20.166657 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:46561
--local_ip_for_outbound_sockets=127.25.240.66
--tserver_master_addrs=127.25.240.126:43969
--webserver_port=37269
--webserver_interface=127.25.240.66
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:20.226611 9083 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:40:20.166146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:59705)
0924 13:40:20.166532 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.166576 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:20.166743 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:20.226390 (+ 59647us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":101,"threads_started":1}
W20260924 13:40:20.226868 9080 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:43969 (0 consecutive failures): Network error: Failed to ping master at 127.25.240.126:43969: Client connection negotiation failed: client connection to 127.25.240.126:43969: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:40:20.227239 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:20.227073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:55529)
0924 13:40:20.227139 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.227205 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:40:20.227607 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:20.227480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:32839)
0924 13:40:20.227559 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.227585 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 13:40:20.227725 9080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:40:20.250387 9084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:20.250581 9084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:20.250603 9084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:20.251777 9084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:20.251820 9084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:20.251847 9084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:40:20.253207 9084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:46561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=37269
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:20.253383 9084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:20.253600 9084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:20.258625 9090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:20.258648 9089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:20.258622 9092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:20.258912 9084 server_base.cc:1034] running on GCE node
I20260924 13:40:20.259140 9084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:20.259383 9084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:20.260527 9084 hybrid_clock.cc:648] HybridClock initialized: now 1790257220260505 us; error 37 us; skew 500 ppm
Sep 24 13:40:20 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257220, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:20.262976 9084 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:40:20.263384 9084 webserver.cc:460] Webserver started at http://127.25.240.66:37269/ using document root <none> and password file <none>
I20260924 13:40:20.263609 9084 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:20.263676 9084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:20.264985 9084 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:20.265733 9099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:20.265901 9084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:20.265997 9084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "9e8d680a8ab7454b8812fc922a88a29b"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "f089481274238d8dcc0492342a766907"
server_key_iv: "ea959b0eab8304a946e14142da3dff7d"
server_key_version: "encryptionkey@0"
I20260924 13:40:20.266336 9084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:20.284452 9084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:20.286811 9084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:20.286974 9084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:20.287207 9084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:20.287652 9106 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 13:40:20.288995 9106 ts_tablet_manager.cc:1852] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 13:40:20.289263 9106 ts_tablet_manager.cc:1905] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:20.291365 9106 ts_tablet_manager.cc:1918] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:20.291452 9106 log.cc:1196] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:20.291661 9106 ts_tablet_manager.cc:1939] T ef05f011cbb14d909b63d4a3143c1101 P 9e8d680a8ab7454b8812fc922a88a29b: Deleting consensus metadata
I20260924 13:40:20.291829 9084 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 13:40:20.291891 9084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 13:40:20.291935 9084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:20.291962 9084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:20.297799 9084 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:46561
I20260924 13:40:20.297855 9214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:46561 every 8 connection(s)
I20260924 13:40:20.298131 9084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:40:20.299055 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:20.298700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:37235)
0924 13:40:20.298855 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.298922 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
W20260924 13:40:20.299265 9215 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:43969 (0 consecutive failures): Network error: Failed to ping master at 127.25.240.126:43969: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
I20260924 13:40:20.299535 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:20.299440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:60945)
0924 13:40:20.299488 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.299513 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 13:40:20.299803 9217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:20.299722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:60327)
0924 13:40:20.299763 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.299784 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:40:20.299897 9215 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:40:20.301631 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 9084
I20260924 13:40:20.301872 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:42935
--local_ip_for_outbound_sockets=127.25.240.67
--tserver_master_addrs=127.25.240.126:43969
--webserver_port=41265
--webserver_interface=127.25.240.67
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:40909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:20.383739 9218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:20.383960 9218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:20.383983 9218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:20.385082 9218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:20.385119 9218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:20.385145 9218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:40:20.386708 9218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:40909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:42935
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=41265
--tserver_master_addrs=127.25.240.126:43969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:20.386901 9218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:20.387075 9218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:20.392148 9224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:20.392141 9226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:20.392215 9218 server_base.cc:1034] running on GCE node
W20260924 13:40:20.392169 9223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:20.392541 9218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:20.392786 9218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:20.393942 9218 hybrid_clock.cc:648] HybridClock initialized: now 1790257220393917 us; error 39 us; skew 500 ppm
Sep 24 13:40:20 dist-test-slave-m5z1 krb5kdc[7781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257220, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:20.396401 9218 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:40:20.396862 9218 webserver.cc:460] Webserver started at http://127.25.240.67:41265/ using document root <none> and password file <none>
I20260924 13:40:20.397050 9218 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:20.397094 9218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:20.398458 9218 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:20.399025 9233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:20.399194 9218 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:20.399298 9218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "2d0770d0636e41a4aff7ab3540f56ff7"
format_stamp: "Formatted at 2026-09-24 13:40:18 on dist-test-slave-m5z1"
server_key: "7cc1d874ef5889ec218305c32d8e290e"
server_key_iv: "35e2f1caa7deb92c035e592e3c628257"
server_key_version: "encryptionkey@0"
I20260924 13:40:20.399636 9218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:20.420619 9218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:20.422680 9218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:20.422847 9218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:20.423086 9218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:20.423411 9218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:20.423466 9218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:20.423508 9218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:20.423537 9218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:20.429514 9218 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:42935
I20260924 13:40:20.429593 9346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:42935 every 8 connection(s)
I20260924 13:40:20.429927 9218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:40:20.430950 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:20.430585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:57589)
0924 13:40:20.430731 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.430810 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
W20260924 13:40:20.431385 9347 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:43969 (0 consecutive failures): Network error: Failed to ping master at 127.25.240.126:43969: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
I20260924 13:40:20.431675 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:20.431587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:49229)
0924 13:40:20.431633 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.431659 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 13:40:20.431975 9349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:20.431902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:43235)
0924 13:40:20.431941 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:20.431962 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:40:20.432228 9347 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:40:20.436793 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 9218
I20260924 13:40:21.228330 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:21.228059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:48203)
0924 13:40:21.228192 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:21.228235 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260924 13:40:21.300413 9352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:21.300213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:57741)
0924 13:40:21.300345 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:21.300381 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260924 13:40:21.432682 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:21.432510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:51927)
0924 13:40:21.432610 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:21.432638 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":53,"threads_started":1}
I20260924 13:40:21.870529 9354 raft_consensus.cc:482] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:21.870644 9354 raft_consensus.cc:504] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:21.871055 9354 leader_election.cc:288] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561), 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935)
Sep 24 13:40:21 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257220, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.67@KRBTEST.COM
Sep 24 13:40:21 dist-test-slave-m5z1 krb5kdc[7781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257220, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.66@KRBTEST.COM
I20260924 13:40:21.881145 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:21.871323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:51629 (local address 127.25.240.67:42935)
0924 13:40:21.871494 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:21.871513 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:21.871543 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:40:21.871984 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:21.871985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:21.872025 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:21.872135 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:21.872917 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:21.873914 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:21.874943 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:21.875334 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:21.875451 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:21.877710 (+ 2259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:21.877714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:21.877714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:21.877726 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:21.879427 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:21.879850 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:21.879853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:21.879853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:21.879943 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:21.880205 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:21.880206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:21.880206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:21.880532 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:21.880617 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:21.880956 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 13:40:21.881067 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":54,"threads_started":1}
I20260924 13:40:21.881165 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:21.871228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:42935 (local address 127.25.240.65:51629)
0924 13:40:21.871372 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:21.871382 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:21.871439 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:21.872200 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:21.872203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:21.872594 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:21.872793 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:21.872798 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:21.874001 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:21.874002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:21.874841 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:21.874844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:21.875425 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:21.875426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:21.875548 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:21.876118 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:21.876126 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:21.877602 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:21.879532 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:21.879539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:21.879541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:21.879763 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:21.880016 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:21.880016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:21.880017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:21.880127 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:21.880612 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:21.880618 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:21.880771 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 13:40:21.880874 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 13:40:21.881059 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260924 13:40:21.881661 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:21.871184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46561 (local address 127.25.240.65:50019)
0924 13:40:21.871336 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:21.871348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:21.871409 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:21.872218 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:21.872219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:21.872593 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:21.873029 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:21.873032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:21.874172 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:21.874173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:21.874840 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:21.874842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:21.875426 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:21.875427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:21.875550 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:21.876174 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:21.876178 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:21.878138 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:21.880301 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:21.880302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:21.880302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:21.880493 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:21.880773 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:21.880773 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:21.880774 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:21.880836 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:21.881325 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:21.881326 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:21.881368 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 13:40:21.881432 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:40:21.881579 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":7,"thread_start_us":55,"threads_started":1}
W20260924 13:40:21.881999 8965 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:21.882179 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:21.871311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:50019 (local address 127.25.240.66:46561)
0924 13:40:21.871536 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:21.871553 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:21.871571 (+ 18us) server_negotiation.cc:407] Connection header received
0924 13:40:21.872033 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:21.872035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:21.872070 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:21.872220 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:21.873133 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:21.874082 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:21.874953 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:21.875334 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:21.875447 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:21.878236 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:21.878241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:21.878242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:21.878252 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:21.880230 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:21.880593 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:21.880596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:21.880596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:21.880698 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:21.880924 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:21.880925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:21.880925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:21.881208 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:21.881293 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:21.881897 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 13:40:21.882019 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
W20260924 13:40:21.882751 8966 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:21.882817 8966 leader_election.cc:302] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9556162d853848a99bb9ec68359452a4; no voters: 2d0770d0636e41a4aff7ab3540f56ff7, 9e8d680a8ab7454b8812fc922a88a29b
I20260924 13:40:21.882926 9354 raft_consensus.cc:2726] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:40:22.228899 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:22.228727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:38667)
0924 13:40:22.228801 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:22.228836 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 13:40:22.300980 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:22.300799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:34909)
0924 13:40:22.300916 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:22.300945 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260924 13:40:22.433297 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:22.433093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:50635)
0924 13:40:22.433222 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:22.433260 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":56,"threads_started":1}
I20260924 13:40:23.229681 9361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:23.229374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:58097)
0924 13:40:23.229520 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:23.229561 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260924 13:40:23.301654 9362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:23.301403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:32961)
0924 13:40:23.301576 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:23.301615 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
I20260924 13:40:23.433971 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:23.433781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:53341)
0924 13:40:23.433894 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:23.433930 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260924 13:40:24.059943 9364 raft_consensus.cc:482] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:24.060026 9364 raft_consensus.cc:504] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:24.060225 9364 leader_election.cc:288] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561), 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935)
W20260924 13:40:24.060617 8966 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
W20260924 13:40:24.060715 8965 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:24.060809 8965 leader_election.cc:302] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9556162d853848a99bb9ec68359452a4; no voters: 2d0770d0636e41a4aff7ab3540f56ff7, 9e8d680a8ab7454b8812fc922a88a29b
I20260924 13:40:24.060885 9364 raft_consensus.cc:2726] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:40:24.230353 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:24.230141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:42709)
0924 13:40:24.230256 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:24.230291 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260924 13:40:24.302335 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:24.302111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:43109)
0924 13:40:24.302252 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:24.302293 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260924 13:40:24.434593 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:24.434391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:39771)
0924 13:40:24.434517 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:24.434553 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260924 13:40:25.231005 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:25.230809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:40471)
0924 13:40:25.230925 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:25.230959 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260924 13:40:25.302978 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:25.302792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:37133)
0924 13:40:25.302905 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:25.302939 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260924 13:40:25.435256 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:25.435039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:35865)
0924 13:40:25.435176 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:25.435215 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260924 13:40:26.077745 9371 raft_consensus.cc:482] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:26.077862 9371 raft_consensus.cc:504] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:26.078061 9371 leader_election.cc:288] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561), 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935)
W20260924 13:40:26.078368 8966 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
W20260924 13:40:26.078444 8965 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:26.078526 8965 leader_election.cc:302] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9556162d853848a99bb9ec68359452a4; no voters: 2d0770d0636e41a4aff7ab3540f56ff7, 9e8d680a8ab7454b8812fc922a88a29b
I20260924 13:40:26.078608 9371 raft_consensus.cc:2726] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:40:26.231711 9372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:26.231485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:41927)
0924 13:40:26.231617 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:26.231657 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260924 13:40:26.303784 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:26.303480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:55573)
0924 13:40:26.303695 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:26.303739 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260924 13:40:26.435923 9374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:26.435721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:44581)
0924 13:40:26.435843 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:26.435880 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260924 13:40:27.232419 9375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:27.232207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:47063)
0924 13:40:27.232339 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:27.232374 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260924 13:40:27.304555 9376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:27.304329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:55709)
0924 13:40:27.304457 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:27.304501 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260924 13:40:27.436831 9377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:27.436600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:52499)
0924 13:40:27.436759 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:27.436791 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260924 13:40:28.233176 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:28.232881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:36275)
0924 13:40:28.233089 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:28.233127 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":59,"threads_started":1}
I20260924 13:40:28.305150 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:28.304947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:46271)
0924 13:40:28.305073 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:28.305109 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
I20260924 13:40:28.409399 9380 raft_consensus.cc:482] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:28.409488 9380 raft_consensus.cc:504] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:28.409722 9380 leader_election.cc:288] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561), 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935)
W20260924 13:40:28.410101 8966 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
W20260924 13:40:28.410205 8965 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:28.410272 8965 leader_election.cc:302] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9556162d853848a99bb9ec68359452a4; no voters: 2d0770d0636e41a4aff7ab3540f56ff7, 9e8d680a8ab7454b8812fc922a88a29b
I20260924 13:40:28.410359 9380 raft_consensus.cc:2726] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:40:28.437762 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:28.437392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:44827)
0924 13:40:28.437584 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:28.437695 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260924 13:40:29.233956 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:29.233712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:52739)
0924 13:40:29.233864 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:29.233908 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260924 13:40:29.305940 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:29.305734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:38811)
0924 13:40:29.305864 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:29.305901 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260924 13:40:29.438459 9384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:29.438224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:40879)
0924 13:40:29.438372 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:29.438411 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":59,"threads_started":1}
I20260924 13:40:30.234684 9385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:30.234447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:59767)
0924 13:40:30.234602 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:30.234637 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260924 13:40:30.306550 9386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:30.306352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:42143)
0924 13:40:30.306475 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:30.306512 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260924 13:40:30.439149 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:30.438933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:56309)
0924 13:40:30.439064 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:30.439106 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260924 13:40:31.235317 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:31.235090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:33261)
0924 13:40:31.235234 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:31.235272 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260924 13:40:31.307190 9389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:31.306979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:46151)
0924 13:40:31.307115 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:31.307149 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260924 13:40:31.439891 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:31.439656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:52639)
0924 13:40:31.439805 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:31.439848 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260924 13:40:32.236119 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:32.235849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:57989)
0924 13:40:32.236027 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:32.236059 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":112,"threads_started":1}
I20260924 13:40:32.307849 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:32.307641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:33327)
0924 13:40:32.307770 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:32.307808 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260924 13:40:32.440627 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:32.440354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:46493)
0924 13:40:32.440488 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:32.440559 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260924 13:40:33.236747 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:33.236526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:44631)
0924 13:40:33.236664 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:33.236701 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260924 13:40:33.308468 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:33.308245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:48557)
0924 13:40:33.308363 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:33.308402 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":73,"threads_started":1}
I20260924 13:40:33.441373 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:33.441149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:50403)
0924 13:40:33.441280 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:33.441325 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260924 13:40:34.237452 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:34.237226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:50999)
0924 13:40:34.237370 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:34.237405 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260924 13:40:34.309113 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:34.308892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:35033)
0924 13:40:34.309028 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:34.309069 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260924 13:40:34.442147 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:34.441915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:58717)
0924 13:40:34.442037 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:34.442074 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260924 13:40:35.225493 9400 raft_consensus.cc:482] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:35.225623 9400 raft_consensus.cc:504] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e8d680a8ab7454b8812fc922a88a29b" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46561 } } peers { permanent_uuid: "2d0770d0636e41a4aff7ab3540f56ff7" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 42935 } } peers { permanent_uuid: "9556162d853848a99bb9ec68359452a4" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 38447 } }
I20260924 13:40:35.225843 9400 leader_election.cc:288] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561), 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935)
W20260924 13:40:35.226224 8965 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2d0770d0636e41a4aff7ab3540f56ff7 (127.25.240.67:42935): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
W20260924 13:40:35.226279 8966 leader_election.cc:341] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9e8d680a8ab7454b8812fc922a88a29b (127.25.240.66:46561): Not found: Tablet not found: ef05f011cbb14d909b63d4a3143c1101
I20260924 13:40:35.226305 8965 leader_election.cc:302] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9556162d853848a99bb9ec68359452a4; no voters: 2d0770d0636e41a4aff7ab3540f56ff7, 9e8d680a8ab7454b8812fc922a88a29b
I20260924 13:40:35.226363 9400 raft_consensus.cc:2726] T ef05f011cbb14d909b63d4a3143c1101 P 9556162d853848a99bb9ec68359452a4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:40:35.238175 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:35.237994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.65:38163)
0924 13:40:35.238106 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:35.238139 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260924 13:40:35.309939 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:35.309704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.66:34685)
0924 13:40:35.309857 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:35.309896 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260924 13:40:35.442482 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 8920
I20260924 13:40:35.443238 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:35.442712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:43969 (local address 127.25.240.67:60385)
0924 13:40:35.443124 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:35.443169 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:43969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":52,"threads_started":1}
I20260924 13:40:35.447419 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 9084
I20260924 13:40:35.451911 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 9218
2026-09-24T13:40:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18452 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:40:35 dist-test-slave-m5z1 krb5kdc[9406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:40:35 dist-test-slave-m5z1 krb5kdc[9406](info): set up 2 sockets
Sep 24 13:40:35 dist-test-slave-m5z1 krb5kdc[9406](info): commencing operation
krb5kdc: starting...
W20260924 13:40:36.545887 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:40:36 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:40:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:40:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:40:36.664705 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39185
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:36.747385 9419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:36.747591 9419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:36.747613 9419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:36.748771 9419 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:40:36.748812 9419 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:40:36.748826 9419 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:40:36.748837 9419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:36.748850 9419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:36.748903 9419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:36.748919 9419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:36.748941 9419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:36.750468 9419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:36.750728 9419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:36.750963 9419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:36.756121 9427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:36.756227 9419 server_base.cc:1034] running on GCE node
W20260924 13:40:36.756100 9425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:36.756096 9424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:36.756559 9419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:36.756851 9419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:36.758013 9419 hybrid_clock.cc:648] HybridClock initialized: now 1790257236757992 us; error 52 us; skew 500 ppm
Sep 24 13:40:36 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257236, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:36.760350 9419 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:36.760824 9419 webserver.cc:460] Webserver started at http://127.25.240.126:39027/ using document root <none> and password file <none>
I20260924 13:40:36.761044 9419 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:36.761111 9419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:36.761264 9419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:36.762359 9419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "696fa76b72764e84bc9fab5f125af409"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "922e10ae863f1478471ef50780d4801c"
server_key_iv: "499940cb8529bad45b8e7e60195f47d1"
server_key_version: "encryptionkey@0"
I20260924 13:40:36.762722 9419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "696fa76b72764e84bc9fab5f125af409"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "922e10ae863f1478471ef50780d4801c"
server_key_iv: "499940cb8529bad45b8e7e60195f47d1"
server_key_version: "encryptionkey@0"
I20260924 13:40:36.764008 9419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:40:36.764667 9434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:36.764835 9419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:40:36.764937 9419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "696fa76b72764e84bc9fab5f125af409"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "922e10ae863f1478471ef50780d4801c"
server_key_iv: "499940cb8529bad45b8e7e60195f47d1"
server_key_version: "encryptionkey@0"
I20260924 13:40:36.765008 9419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:36.785126 9419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:36.786947 9419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:36.787108 9419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:36.791165 9419 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:39185
I20260924 13:40:36.791201 9486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:39185 every 8 connection(s)
I20260924 13:40:36.791548 9419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:36.792395 9487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:36.794595 9487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Bootstrap starting.
I20260924 13:40:36.795111 9487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:36.795351 9487 log.cc:824] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:36.795953 9487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: No bootstrap required, opened a new log
I20260924 13:40:36.797441 9487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:40:36.797616 9487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:36.797664 9487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696fa76b72764e84bc9fab5f125af409, State: Initialized, Role: FOLLOWER
I20260924 13:40:36.797789 9487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:40:36.797847 9487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:40:36.797865 9487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:40:36.797896 9487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:36.798362 9487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:40:36.798442 9487 leader_election.cc:302] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 696fa76b72764e84bc9fab5f125af409; no voters:
I20260924 13:40:36.798565 9487 leader_election.cc:288] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:40:36.798629 9490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:36.798794 9487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:36.798810 9490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 1 LEADER]: Becoming Leader. State: Replica: 696fa76b72764e84bc9fab5f125af409, State: Running, Role: LEADER
I20260924 13:40:36.798888 9490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:40:36.799073 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 9419
I20260924 13:40:36.799180 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:40:36.799425 26561 external_mini_cluster.cc:1428] Setting key b8043a84ac153e526d34df2daafeaa36
I20260924 13:40:36.799983 9492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 696fa76b72764e84bc9fab5f125af409. Latest consensus state: current_term: 1 leader_uuid: "696fa76b72764e84bc9fab5f125af409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } } }
I20260924 13:40:36.800086 9492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:36.800371 9491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "696fa76b72764e84bc9fab5f125af409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } } }
I20260924 13:40:36.800626 9491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:36.800302 9505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:36.801115 9505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:36.802618 9505 catalog_manager.cc:1347] Generated new cluster ID: 86590024d35b481fbf96c4cf25550d33
I20260924 13:40:36.802670 9505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:40:36 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:36.810254 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:36.800097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.25.240.126:39185)
0924 13:40:36.800348 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:36.800357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:36.800392 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:40:36.800583 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:36.800590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:36.800648 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:36.800783 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:36.801471 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:36.802920 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:36.803687 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:36.804108 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:36.804235 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:36.806537 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:36.806541 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:36.806541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:36.806553 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:36.808671 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:36.809055 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:36.809058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:36.809058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:36.809148 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:36.809362 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:36.809362 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:36.809363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:36.809663 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:36.809823 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:36.809918 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 13:40:36.810056 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":106,"threads_started":1}
I20260924 13:40:36.819801 9505 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:40:36.820497 9505 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:40:36.826773 9505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Generated new TSK 0
I20260924 13:40:36.826968 9505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:40:36.858394 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39185
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:36.950011 9513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:36.950214 9513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:36.950244 9513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:36.951406 9513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:36.951457 9513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:36.951483 9513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:40:36.953081 9513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:36.953346 9513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:36.953619 9513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:36.958613 9518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:36.958616 9521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:36.958734 9513 server_base.cc:1034] running on GCE node
W20260924 13:40:36.958654 9519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:36.959074 9513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:36.959321 9513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:36.960481 9513 hybrid_clock.cc:648] HybridClock initialized: now 1790257236960475 us; error 36 us; skew 500 ppm
Sep 24 13:40:36 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257236, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:36.962970 9513 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:40:36.963346 9513 webserver.cc:460] Webserver started at http://127.25.240.65:40501/ using document root <none> and password file <none>
I20260924 13:40:36.963580 9513 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:36.963626 9513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:36.963721 9513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:36.964707 9513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "8f6fdd93a38840ada721cefaaa3c64de"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "ce56f7d356a7d0643094dfd1ecf4c404"
server_key_iv: "33b1be60ac7d78bfa3aada3069e743f3"
server_key_version: "encryptionkey@0"
I20260924 13:40:36.964996 9513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "8f6fdd93a38840ada721cefaaa3c64de"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "ce56f7d356a7d0643094dfd1ecf4c404"
server_key_iv: "33b1be60ac7d78bfa3aada3069e743f3"
server_key_version: "encryptionkey@0"
I20260924 13:40:36.966360 9513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:40:36.967147 9528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:36.967341 9513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:36.967432 9513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "8f6fdd93a38840ada721cefaaa3c64de"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "ce56f7d356a7d0643094dfd1ecf4c404"
server_key_iv: "33b1be60ac7d78bfa3aada3069e743f3"
server_key_version: "encryptionkey@0"
I20260924 13:40:36.967489 9513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:36.986968 9513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:36.988807 9513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:36.988922 9513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:36.989109 9513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:36.989449 9513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:36.989483 9513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:36.989531 9513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:36.989580 9513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:36.995357 9513 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:35903
I20260924 13:40:36.995437 9641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:35903 every 8 connection(s)
I20260924 13:40:36.995764 9513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:40:37 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257236, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:37.004156 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 9513
I20260924 13:40:37.004273 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:40:37.004465 26561 external_mini_cluster.cc:1428] Setting key e47cddf97c8dfa4e1abef5fbc6deee2e
I20260924 13:40:37.005014 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:36.996517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:33993 (local address 127.25.240.126:39185)
0924 13:40:36.996583 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:36.996584 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:36.996791 (+ 207us) server_negotiation.cc:407] Connection header received
0924 13:40:36.997265 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:36.997266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:36.997278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:36.997337 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:36.998331 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:36.998872 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:36.999950 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.000210 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.000274 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.002414 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:37.002415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:37.002416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:37.002421 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:37.003642 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:37.003989 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:37.003990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:37.003990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:37.004053 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:37.004294 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:37.004295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:37.004295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:37.004432 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:37.004511 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.004838 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.004925 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:40:37.005213 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:36.996473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:33993)
0924 13:40:36.996652 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:36.996702 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:36.997175 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:36.997406 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:36.997409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:36.998058 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:36.998252 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:36.998257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:36.998974 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:36.998975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:36.999872 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:36.999875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.000287 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.000288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.000390 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.000907 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:37.000923 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:37.002337 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:37.003709 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:37.003715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:37.003717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:37.003919 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:37.004118 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:37.004119 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:37.004119 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:37.004213 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:37.004503 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:37.004509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:37.004662 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 13:40:37.004813 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 13:40:37.004965 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260924 13:40:37.005661 9642 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39185
I20260924 13:40:37.005793 9642 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:37.005964 9642 heartbeater.cc:511] Master 127.25.240.126:39185 requested a full tablet report, sending...
I20260924 13:40:37.006512 9451 ts_manager.cc:194] Registered new tserver with Master: 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903)
I20260924 13:40:37.007431 9451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:33993
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:40:37.028460 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39185
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:37.108156 9647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:37.108381 9647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:37.108417 9647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:37.109534 9647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:37.109623 9647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:37.109660 9647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:40:37.111048 9647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9647
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:37.111219 9647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:37.111382 9647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:37.116248 9652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:37.116290 9653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:37.116261 9655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:37.116454 9647 server_base.cc:1034] running on GCE node
I20260924 13:40:37.116683 9647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:37.116918 9647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:37.118111 9647 hybrid_clock.cc:648] HybridClock initialized: now 1790257237118094 us; error 56 us; skew 500 ppm
Sep 24 13:40:37 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257237, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:37.120610 9647 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:40:37.121019 9647 webserver.cc:460] Webserver started at http://127.25.240.66:33409/ using document root <none> and password file <none>
I20260924 13:40:37.121243 9647 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:37.121287 9647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:37.121428 9647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:37.122500 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "cb198887250c4e85a2ce5b788da38c36"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "3b6613221a98dfd6ad1dce07359c3134"
server_key_iv: "3d02673c8a45632acb3e8cbdfcee5f90"
server_key_version: "encryptionkey@0"
I20260924 13:40:37.122848 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "cb198887250c4e85a2ce5b788da38c36"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "3b6613221a98dfd6ad1dce07359c3134"
server_key_iv: "3d02673c8a45632acb3e8cbdfcee5f90"
server_key_version: "encryptionkey@0"
I20260924 13:40:37.124127 9647 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:40:37.124792 9662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:37.124960 9647 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:37.125039 9647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "cb198887250c4e85a2ce5b788da38c36"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "3b6613221a98dfd6ad1dce07359c3134"
server_key_iv: "3d02673c8a45632acb3e8cbdfcee5f90"
server_key_version: "encryptionkey@0"
I20260924 13:40:37.125089 9647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:37.145316 9647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:37.147643 9647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:37.147756 9647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:37.147934 9647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:37.148263 9647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:37.148298 9647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:37.148319 9647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:37.148368 9647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:37.154158 9647 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:46009
I20260924 13:40:37.154201 9775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:46009 every 8 connection(s)
I20260924 13:40:37.154518 9647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:40:37 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257237, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:37.162959 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 9647
I20260924 13:40:37.163062 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:40:37.163288 26561 external_mini_cluster.cc:1428] Setting key 114c390830b2f5fc8737e42d1fb61b1e
I20260924 13:40:37.163589 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.155270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:47967 (local address 127.25.240.126:39185)
0924 13:40:37.155340 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.155342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.155578 (+ 236us) server_negotiation.cc:407] Connection header received
0924 13:40:37.156103 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.156104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.156117 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.156165 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:37.157101 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.157708 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.158695 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.158908 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.158985 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.160979 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:37.160981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:37.160981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:37.160988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:37.162268 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:37.162608 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:37.162609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:37.162609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:37.162656 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:37.162879 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:37.162880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:37.162880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:37.163028 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:37.163101 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.163407 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.163495 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:40:37.163760 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.155262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:47967)
0924 13:40:37.155417 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.155479 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:37.155994 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:37.156261 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:37.156266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:37.156814 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:37.156998 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.157003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.157802 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.157804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.158595 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.158597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.158990 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.158992 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.159118 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.159541 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:37.159549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:37.160893 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:37.162336 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:37.162342 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:37.162343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:37.162546 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:37.162722 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:37.162722 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:37.162723 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:37.162815 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:37.163114 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:37.163120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:37.163252 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 13:40:37.163367 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 13:40:37.163556 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260924 13:40:37.164219 9776 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39185
I20260924 13:40:37.164321 9776 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:37.164503 9776 heartbeater.cc:511] Master 127.25.240.126:39185 requested a full tablet report, sending...
I20260924 13:40:37.164858 9451 ts_manager.cc:194] Registered new tserver with Master: cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009)
I20260924 13:40:37.165221 9451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:47967
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:40:37.187983 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39185
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:37.267939 9781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:37.268141 9781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:37.268163 9781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:37.269337 9781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:37.269381 9781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:37.269408 9781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:40:37.270953 9781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:37.271199 9781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:37.271420 9781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:37.276672 9786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:37.276657 9789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:37.276883 9781 server_base.cc:1034] running on GCE node
W20260924 13:40:37.276657 9787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:37.277159 9781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:37.277407 9781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:37.278569 9781 hybrid_clock.cc:648] HybridClock initialized: now 1790257237278557 us; error 30 us; skew 500 ppm
Sep 24 13:40:37 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257237, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:37.281062 9781 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:40:37.281499 9781 webserver.cc:460] Webserver started at http://127.25.240.67:35523/ using document root <none> and password file <none>
I20260924 13:40:37.281816 9781 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:37.281901 9781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:37.282058 9781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:40:37.283360 9781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "20a275cf02194c7780533e64d5a4ff98"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "c58ea469ef6bdc817acebc176ce24486"
server_key_iv: "75d89281f9e59170381aabd8d18e3c04"
server_key_version: "encryptionkey@0"
I20260924 13:40:37.283753 9781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "20a275cf02194c7780533e64d5a4ff98"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "c58ea469ef6bdc817acebc176ce24486"
server_key_iv: "75d89281f9e59170381aabd8d18e3c04"
server_key_version: "encryptionkey@0"
I20260924 13:40:37.285454 9781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 13:40:37.286322 9796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:37.286491 9781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:37.286586 9781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "20a275cf02194c7780533e64d5a4ff98"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "c58ea469ef6bdc817acebc176ce24486"
server_key_iv: "75d89281f9e59170381aabd8d18e3c04"
server_key_version: "encryptionkey@0"
I20260924 13:40:37.286674 9781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:37.296890 9781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:37.299170 9781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:37.299328 9781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:37.299597 9781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:37.299898 9781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:37.299952 9781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:37.299993 9781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:37.300020 9781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:37.306041 9781 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:34585
I20260924 13:40:37.306108 9910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:34585 every 8 connection(s)
I20260924 13:40:37.306413 9781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:40:37.312374 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 9781
I20260924 13:40:37.312472 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:40:37.312668 26561 external_mini_cluster.cc:1428] Setting key efa48e43c541f6ab50e4963d46c86eac
Sep 24 13:40:37 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257237, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:37.317301 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.307496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:38267 (local address 127.25.240.126:39185)
0924 13:40:37.307578 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.307580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.307591 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:40:37.308025 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.308026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.308039 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.308093 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:37.309251 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.309838 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.311254 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.311449 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.311531 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.314276 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:37.314278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:37.314278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:37.314287 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:37.315844 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:37.316302 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:37.316303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:37.316303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:37.316366 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:37.316620 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:37.316621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:37.316621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:37.316743 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:37.316822 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.317122 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.317202 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 13:40:37.317495 9913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.307191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:38267)
0924 13:40:37.307341 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.307381 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:37.307920 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:37.308162 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:37.308166 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:37.308901 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:37.309155 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.309160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.309928 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.309929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.311154 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.311156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.311540 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.311542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.311679 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.312202 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:37.312213 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:37.314184 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:37.315948 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:37.315955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:37.315957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:37.316206 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:37.316433 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:37.316434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:37.316434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:37.316562 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:37.316813 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:37.316819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:37.316955 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 13:40:37.317096 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 13:40:37.317248 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260924 13:40:37.317973 9911 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39185
I20260924 13:40:37.318074 9911 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:37.318250 9911 heartbeater.cc:511] Master 127.25.240.126:39185 requested a full tablet report, sending...
I20260924 13:40:37.318861 9451 ts_manager.cc:194] Registered new tserver with Master: 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585)
I20260924 13:40:37.319416 9451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:38267
I20260924 13:40:37.324525 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:40:37.333339 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.325775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.25.240.126:39185)
0924 13:40:37.325829 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.325831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.325940 (+ 109us) server_negotiation.cc:407] Connection header received
0924 13:40:37.326015 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.326016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.326026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.326080 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:37.326628 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.327197 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.328008 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.328198 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.328297 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.329744 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:37.329745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:37.329746 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:37.329753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:37.331705 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:37.332122 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:37.332123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:37.332123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:37.332185 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:37.332411 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:37.332412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:37.332412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:37.332868 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:37.332991 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.333124 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.333223 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:40:37.335424 9451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:40:37.337911 9451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:40:37.345785 9922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.340398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:35903 (local address 127.0.0.1:33152)
0924 13:40:37.340809 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.340853 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:37.340957 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:37.341385 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:37.341388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:37.341390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:37.341457 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.341460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.342965 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.342966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.344543 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.344544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.345466 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.345467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.345547 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.345581 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 13:40:37.345655 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 13:40:37.345702 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":61,"thread_start_us":51,"threads_started":1}
I20260924 13:40:37.345839 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.340641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.25.240.65:35903)
0924 13:40:37.340824 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.340828 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.340916 (+ 88us) server_negotiation.cc:407] Connection header received
0924 13:40:37.341083 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.341086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.341103 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.341287 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:37.341679 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.342840 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.344641 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.345387 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.345467 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.345530 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.345668 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.345735 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260924 13:40:37.346659 9923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.340399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34585 (local address 127.0.0.1:36516)
0924 13:40:37.341103 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.341113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:37.341166 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:37.341438 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:37.341439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:37.341442 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:37.341498 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.341500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.343130 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.343131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.344541 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.344544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.346427 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.346428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.346500 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.346504 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:40:37.346556 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 13:40:37.346591 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":5,"spinlock_wait_cycles":896,"thread_start_us":86,"threads_started":1}
I20260924 13:40:37.346858 9576 tablet_service.cc:1467] Processing CreateTablet for tablet 87bd8e4446794ee7bd7fade4f1c042eb (DEFAULT_TABLE table=test-table [id=ae2a82786c974610af99528009a7b551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:37.347045 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.340633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36516 (local address 127.25.240.67:34585)
0924 13:40:37.340916 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.340919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.341187 (+ 268us) server_negotiation.cc:407] Connection header received
0924 13:40:37.341237 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.341240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.341267 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.341356 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:37.341634 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.343021 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.345624 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.346357 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.346428 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.346479 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.346912 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.346951 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":71,"threads_started":1}
I20260924 13:40:37.347265 9576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bd8e4446794ee7bd7fade4f1c042eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:37.347427 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.340399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46009 (local address 127.0.0.1:55342)
0924 13:40:37.341241 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.341250 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:37.341289 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:37.343511 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:37.343512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:37.343514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:37.343559 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.343560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.344866 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.344867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.346090 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.346093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.347189 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.347190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.347263 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.347267 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:40:37.347329 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 13:40:37.347362 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":5,"thread_start_us":89,"threads_started":1}
I20260924 13:40:37.347661 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.340641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.25.240.66:46009)
0924 13:40:37.340996 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.341000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.343275 (+ 2275us) server_negotiation.cc:407] Connection header received
0924 13:40:37.343325 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.343328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.343347 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.343429 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:37.343633 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.344767 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.346188 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.346982 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.347048 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.347515 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.347544 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.347584 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":75,"threads_started":1}
I20260924 13:40:37.348230 9710 tablet_service.cc:1467] Processing CreateTablet for tablet 87bd8e4446794ee7bd7fade4f1c042eb (DEFAULT_TABLE table=test-table [id=ae2a82786c974610af99528009a7b551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:37.348584 9710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bd8e4446794ee7bd7fade4f1c042eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:37.350449 9845 tablet_service.cc:1467] Processing CreateTablet for tablet 87bd8e4446794ee7bd7fade4f1c042eb (DEFAULT_TABLE table=test-table [id=ae2a82786c974610af99528009a7b551]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:37.350783 9845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bd8e4446794ee7bd7fade4f1c042eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:37.351033 9928 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Bootstrap starting.
I20260924 13:40:37.351683 9928 tablet_bootstrap.cc:654] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:37.352195 9929 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Bootstrap starting.
I20260924 13:40:37.352244 9928 log.cc:824] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:37.352834 9929 tablet_bootstrap.cc:654] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:37.353153 9928 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: No bootstrap required, opened a new log
I20260924 13:40:37.353228 9928 ts_tablet_manager.cc:1397] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:40:37.353452 9929 log.cc:824] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:37.354247 9932 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: Bootstrap starting.
I20260924 13:40:37.354897 9932 tablet_bootstrap.cc:654] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:37.355419 9932 log.cc:824] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:37.355501 9928 raft_consensus.cc:348] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.355726 9928 raft_consensus.cc:374] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:37.355773 9928 raft_consensus.cc:729] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6fdd93a38840ada721cefaaa3c64de, State: Initialized, Role: FOLLOWER
I20260924 13:40:37.355917 9928 consensus_queue.cc:260] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.356169 9932 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: No bootstrap required, opened a new log
I20260924 13:40:37.356233 9642 heartbeater.cc:503] Master 127.25.240.126:39185 was elected leader, sending a full tablet report...
I20260924 13:40:37.356326 9932 ts_tablet_manager.cc:1397] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:40:37.356230 9928 ts_tablet_manager.cc:1428] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:40:37.357254 9929 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: No bootstrap required, opened a new log
I20260924 13:40:37.357326 9929 ts_tablet_manager.cc:1397] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 13:40:37.358594 9932 raft_consensus.cc:348] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.358806 9932 raft_consensus.cc:374] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:37.358852 9932 raft_consensus.cc:729] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a275cf02194c7780533e64d5a4ff98, State: Initialized, Role: FOLLOWER
I20260924 13:40:37.358974 9932 consensus_queue.cc:260] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.359120 9929 raft_consensus.cc:348] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.359264 9911 heartbeater.cc:503] Master 127.25.240.126:39185 was elected leader, sending a full tablet report...
I20260924 13:40:37.359306 9929 raft_consensus.cc:374] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:37.359357 9929 raft_consensus.cc:729] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb198887250c4e85a2ce5b788da38c36, State: Initialized, Role: FOLLOWER
I20260924 13:40:37.359475 9929 consensus_queue.cc:260] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.359251 9932 ts_tablet_manager.cc:1428] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:40:37.359756 9776 heartbeater.cc:503] Master 127.25.240.126:39185 was elected leader, sending a full tablet report...
I20260924 13:40:37.359762 9929 ts_tablet_manager.cc:1428] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 13:40:37.363739 9936 raft_consensus.cc:482] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:37.363866 9936 raft_consensus.cc:504] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.364425 9936 leader_election.cc:288] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903)
I20260924 13:40:37.367931 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.364564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:41503 (local address 127.25.240.67:34585)
0924 13:40:37.364606 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.364608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.364654 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:40:37.364737 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.364738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.364745 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.364827 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:37.365098 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.365876 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.367147 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.367593 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.367680 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.367690 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.367837 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.367872 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:40:37.367949 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.364389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34585 (local address 127.25.240.66:41503)
0924 13:40:37.364595 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.364605 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:37.364668 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:37.364864 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:37.364865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:37.364867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:37.364972 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.364999 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.365967 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.365967 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.367053 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.367055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.367670 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.367671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.367742 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.367746 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:40:37.367829 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 13:40:37.367868 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":9,"thread_start_us":87,"threads_started":1}
I20260924 13:40:37.368297 9778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.364382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:35903 (local address 127.25.240.66:39535)
0924 13:40:37.364424 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.364435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:37.364479 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:37.364688 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:37.364689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:37.364691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:37.364756 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.364758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.365743 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.365743 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.366990 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:37.366993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.367986 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.367988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:37.368088 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.368091 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:40:37.368142 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:40:37.368191 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":23}
I20260924 13:40:37.368263 9865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd8e4446794ee7bd7fade4f1c042eb" candidate_uuid: "cb198887250c4e85a2ce5b788da38c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a275cf02194c7780533e64d5a4ff98" is_pre_election: true
I20260924 13:40:37.368398 9865 raft_consensus.cc:2454] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb198887250c4e85a2ce5b788da38c36 in term 0.
I20260924 13:40:37.368613 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.364492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:39535 (local address 127.25.240.65:35903)
0924 13:40:37.364539 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.364542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.364550 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:40:37.364575 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.364576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.364586 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.364636 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:37.364836 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.365670 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.367100 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.367832 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.367910 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.367919 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.368465 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.368536 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:40:37.368618 9665 leader_election.cc:302] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a275cf02194c7780533e64d5a4ff98, cb198887250c4e85a2ce5b788da38c36; no voters:
I20260924 13:40:37.368794 9936 raft_consensus.cc:2781] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:40:37.368863 9936 raft_consensus.cc:482] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:40:37.368896 9936 raft_consensus.cc:3037] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:37.368860 9596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd8e4446794ee7bd7fade4f1c042eb" candidate_uuid: "cb198887250c4e85a2ce5b788da38c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6fdd93a38840ada721cefaaa3c64de" is_pre_election: true
I20260924 13:40:37.368989 9596 raft_consensus.cc:2454] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb198887250c4e85a2ce5b788da38c36 in term 0.
I20260924 13:40:37.369951 9936 raft_consensus.cc:504] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.370234 9865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd8e4446794ee7bd7fade4f1c042eb" candidate_uuid: "cb198887250c4e85a2ce5b788da38c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a275cf02194c7780533e64d5a4ff98"
I20260924 13:40:37.370342 9865 raft_consensus.cc:3037] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:37.371359 9865 raft_consensus.cc:2454] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb198887250c4e85a2ce5b788da38c36 in term 1.
I20260924 13:40:37.371536 9665 leader_election.cc:302] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a275cf02194c7780533e64d5a4ff98, cb198887250c4e85a2ce5b788da38c36; no voters:
I20260924 13:40:37.371778 9938 raft_consensus.cc:2781] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:37.371773 9936 leader_election.cc:288] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [CANDIDATE]: Term 1 election: Requested vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903)
I20260924 13:40:37.371826 9596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bd8e4446794ee7bd7fade4f1c042eb" candidate_uuid: "cb198887250c4e85a2ce5b788da38c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6fdd93a38840ada721cefaaa3c64de"
I20260924 13:40:37.371899 9596 raft_consensus.cc:3037] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:37.371935 9938 raft_consensus.cc:686] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 LEADER]: Becoming Leader. State: Replica: cb198887250c4e85a2ce5b788da38c36, State: Running, Role: LEADER
I20260924 13:40:37.372017 9938 consensus_queue.cc:237] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:37.372758 9596 raft_consensus.cc:2454] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb198887250c4e85a2ce5b788da38c36 in term 1.
I20260924 13:40:37.372782 9451 catalog_manager.cc:5495] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb198887250c4e85a2ce5b788da38c36 (127.25.240.66). New cstate: current_term: 1 leader_uuid: "cb198887250c4e85a2ce5b788da38c36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } health_report { overall_health: UNKNOWN } } }
I20260924 13:40:37.381294 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.378044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.25.240.66:46009)
0924 13:40:37.378092 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:37.378094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:37.378105 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:40:37.378149 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:37.378149 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:37.378169 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:37.378261 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:40:37.378362 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.379144 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.380283 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:37.380626 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:37.380701 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:37.380863 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:40:37.380996 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:40:37.381094 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:37.381144 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 13:40:37.381169 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 13:40:37.383185 9596 raft_consensus.cc:1261] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Refusing update from remote peer cb198887250c4e85a2ce5b788da38c36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:37.383203 9865 raft_consensus.cc:1261] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 1 FOLLOWER]: Refusing update from remote peer cb198887250c4e85a2ce5b788da38c36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:37.383579 9939 consensus_queue.cc:1035] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:37.383736 9938 consensus_queue.cc:1035] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:37.386876 9944 mvcc.cc:205] Tried to move back new op lower bound from 7332893644318658560 to 7332893644276473856. Current Snapshot: MvccSnapshot[applied={T|T < 7332893644318658560}]
I20260924 13:40:37.388489 9451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:37.388602 9451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:37.389333 9451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Sending DeleteTablet for 3 replicas of tablet 87bd8e4446794ee7bd7fade4f1c042eb
I20260924 13:40:37.389668 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 9513
I20260924 13:40:37.389665 9845 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bd8e4446794ee7bd7fade4f1c042eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:37 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:36516
I20260924 13:40:37.389750 9576 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bd8e4446794ee7bd7fade4f1c042eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:37 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:33152
I20260924 13:40:37.390437 9710 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bd8e4446794ee7bd7fade4f1c042eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:37 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:55342
W20260924 13:40:37.394596 9435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:35903 (error 108)
I20260924 13:40:37.394706 9952 tablet_replica.cc:330] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: stopping tablet replica
I20260924 13:40:37.394814 9952 raft_consensus.cc:2229] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:40:37.394824 9951 tablet_replica.cc:330] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: stopping tablet replica
I20260924 13:40:37.394923 9952 raft_consensus.cc:2258] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:37.395048 9951 raft_consensus.cc:2229] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:37.395200 9951 raft_consensus.cc:2258] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:37.395253 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 9647
I20260924 13:40:37.395339 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.395213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:35903 (local address 127.0.0.1:33158)
0924 13:40:37.395269 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.395305 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 13:40:37.395452 9435 catalog_manager.cc:4584] TS 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87bd8e4446794ee7bd7fade4f1c042eb: Network error: Client connection negotiation failed: client connection to 127.25.240.65:35903: connect: Connection refused (error 111)
I20260924 13:40:37.396097 9951 ts_tablet_manager.cc:1905] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:37.398087 9951 ts_tablet_manager.cc:1918] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:37.398166 9951 log.cc:1196] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/87bd8e4446794ee7bd7fade4f1c042eb
I20260924 13:40:37.398424 9951 ts_tablet_manager.cc:1939] T 87bd8e4446794ee7bd7fade4f1c042eb P 20a275cf02194c7780533e64d5a4ff98: Deleting consensus metadata
I20260924 13:40:37.398870 9437 catalog_manager.cc:4841] TS 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585): tablet 87bd8e4446794ee7bd7fade4f1c042eb (table test-table [id=ae2a82786c974610af99528009a7b551]) successfully deleted
I20260924 13:40:37.401026 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:37.400900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46009 (local address 127.0.0.1:55362)
0924 13:40:37.400961 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:37.400989 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.66:46009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:40:37.401155 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 9781
I20260924 13:40:37.406186 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 9419
I20260924 13:40:57.411531 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39185
--webserver_interface=127.25.240.126
--webserver_port=39027
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:40:57.502439 9954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:57.502635 9954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:57.502657 9954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:57.503782 9954 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:40:57.503822 9954 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:40:57.503835 9954 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:40:57.503846 9954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:40:57.503859 9954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:57.503872 9954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:57.503922 9954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:40:57.503939 9954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:40:57.505383 9954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:57.505645 9954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:57.505891 9954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:57.514580 9961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:57.514557 9960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:57.514776 9954 server_base.cc:1034] running on GCE node
W20260924 13:40:57.514679 9963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:57.515090 9954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:57.515446 9954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:57.516615 9954 hybrid_clock.cc:648] HybridClock initialized: now 1790257257516603 us; error 42 us; skew 500 ppm
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:57.519279 9954 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:40:57.519775 9954 webserver.cc:460] Webserver started at http://127.25.240.126:39027/ using document root <none> and password file <none>
I20260924 13:40:57.520047 9954 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:57.520136 9954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:57.521723 9954 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:40:57.522409 9970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:57.522589 9954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:40:57.522735 9954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "696fa76b72764e84bc9fab5f125af409"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "922e10ae863f1478471ef50780d4801c"
server_key_iv: "499940cb8529bad45b8e7e60195f47d1"
server_key_version: "encryptionkey@0"
I20260924 13:40:57.523147 9954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:57.534509 9954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:57.536726 9954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:57.536887 9954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:57.541069 9954 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:39185
I20260924 13:40:57.541100 10022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:39185 every 8 connection(s)
I20260924 13:40:57.541495 9954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:40:57.543012 10023 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:40:57.543668 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Bootstrap starting.
I20260924 13:40:57.546518 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 9954
I20260924 13:40:57.546834 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:35903
--local_ip_for_outbound_sockets=127.25.240.65
--tserver_master_addrs=127.25.240.126:39185
--webserver_port=40501
--webserver_interface=127.25.240.65
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:57.552394 10023 log.cc:824] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:57.554450 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:40:57.554770 10023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Bootstrap complete.
I20260924 13:40:57.557070 10023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:40:57.557297 10023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696fa76b72764e84bc9fab5f125af409, State: Initialized, Role: FOLLOWER
I20260924 13:40:57.557467 10023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:40:57.557528 10023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:40:57.557606 10023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:40:57.557664 10023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 1 FOLLOWER]: Advancing to term 2
I20260924 13:40:57.558713 10023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:40:57.558797 10023 leader_election.cc:302] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 696fa76b72764e84bc9fab5f125af409; no voters:
I20260924 13:40:57.558970 10023 leader_election.cc:288] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 13:40:57.559070 10027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 13:40:57.559206 10023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:40:57.559273 10027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 2 LEADER]: Becoming Leader. State: Replica: 696fa76b72764e84bc9fab5f125af409, State: Running, Role: LEADER
I20260924 13:40:57.559384 10027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:40:57.559687 10027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 696fa76b72764e84bc9fab5f125af409. Latest consensus state: current_term: 2 leader_uuid: "696fa76b72764e84bc9fab5f125af409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } } }
I20260924 13:40:57.559772 10027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:57.560156 10028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "696fa76b72764e84bc9fab5f125af409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } } }
I20260924 13:40:57.560233 10028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: This master's current role is: LEADER
I20260924 13:40:57.560274 10032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:40:57.560951 10032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:40:57.561028 10032 catalog_manager.cc:1259] Loaded cluster ID: 86590024d35b481fbf96c4cf25550d33
I20260924 13:40:57.561049 10032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:40:57.562107 10032 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:40:57.562288 10032 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Loaded TSK: 0
I20260924 13:40:57.570400 10032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Generated new TSK 1
I20260924 13:40:57.570482 10032 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:40:57.632344 10025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:57.632532 10025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:57.632555 10025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:57.633718 10025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:57.633757 10025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:57.633782 10025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:40:57.635253 10025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:35903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=40501
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:57.635479 10025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:57.635713 10025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:57.640621 10050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:57.640671 10051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:57.640597 10053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:57.640951 10025 server_base.cc:1034] running on GCE node
I20260924 13:40:57.641160 10025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:57.641428 10025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:57.642612 10025 hybrid_clock.cc:648] HybridClock initialized: now 1790257257642592 us; error 32 us; skew 500 ppm
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:57.645504 10025 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:40:57.646001 10025 webserver.cc:460] Webserver started at http://127.25.240.65:40501/ using document root <none> and password file <none>
I20260924 13:40:57.646226 10025 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:57.646306 10025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:57.647639 10025 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:57.648377 10060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:57.648556 10025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:40:57.648653 10025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "8f6fdd93a38840ada721cefaaa3c64de"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "ce56f7d356a7d0643094dfd1ecf4c404"
server_key_iv: "33b1be60ac7d78bfa3aada3069e743f3"
server_key_version: "encryptionkey@0"
I20260924 13:40:57.649008 10025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:57.660430 10025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:57.662714 10025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:57.662882 10025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:57.663106 10025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:57.663539 10067 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:40:57.665362 10025 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:40:57.665423 10025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:40:57.665467 10025 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:40:57.666106 10025 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:40:57.666162 10025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 13:40:57.666240 10067 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Bootstrap starting.
I20260924 13:40:57.673276 10025 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:35903
I20260924 13:40:57.673393 10174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:35903 every 8 connection(s)
I20260924 13:40:57.673735 10025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:40:57.675541 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 10025
I20260924 13:40:57.675836 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:46009
--local_ip_for_outbound_sockets=127.25.240.66
--tserver_master_addrs=127.25.240.126:39185
--webserver_port=33409
--webserver_interface=127.25.240.66
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:57.677136 10067 log.cc:824] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:57.679905 10067 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:40:57.680194 10067 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Bootstrap complete.
I20260924 13:40:57.681501 10067 ts_tablet_manager.cc:1397] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.014s
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:57.682741 10067 raft_consensus.cc:348] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:57.683079 10067 raft_consensus.cc:729] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6fdd93a38840ada721cefaaa3c64de, State: Initialized, Role: FOLLOWER
I20260924 13:40:57.683259 10067 consensus_queue.cc:260] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:57.683549 10067 ts_tablet_manager.cc:1428] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:40:57.686757 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.674604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:59499 (local address 127.25.240.126:39185)
0924 13:40:57.674784 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:57.674793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:57.675006 (+ 213us) server_negotiation.cc:407] Connection header received
0924 13:40:57.675784 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:57.675791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:57.675851 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:57.676194 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:57.677154 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.678286 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.680120 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.680757 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.680888 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.683191 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:57.683196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:57.683196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:57.683207 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:57.684980 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.685420 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.685423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.685423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.685520 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.685832 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.685833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.685834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.686055 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:57.686145 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:57.686449 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 13:40:57.686570 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260924 13:40:57.686756 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.674535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:59499)
0924 13:40:57.674866 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:57.674897 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:57.675454 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:57.676352 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:57.676356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:57.676747 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:57.677011 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.677016 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.678829 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.678831 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.680048 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.680051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.680874 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.680875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.680999 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.681598 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:57.681608 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:57.683051 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:57.685085 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.685092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.685094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.685321 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.685635 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.685636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.685636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.685743 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.686139 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:57.686145 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:57.686275 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 13:40:57.686379 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:40:57.686552 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":91,"threads_started":1}
I20260924 13:40:57.687808 10175 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39185
I20260924 13:40:57.687922 10175 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:57.688097 10175 heartbeater.cc:511] Master 127.25.240.126:39185 requested a full tablet report, sending...
I20260924 13:40:57.688772 9987 ts_manager.cc:194] Registered new tserver with Master: 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903)
I20260924 13:40:57.689934 9987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:59499
I20260924 13:40:57.690891 10175 heartbeater.cc:503] Master 127.25.240.126:39185 was elected leader, sending a full tablet report...
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.65@KRBTEST.COM
I20260924 13:40:57.697743 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.689333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:35903 (local address 127.0.0.1:42108)
0924 13:40:57.689762 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:57.689795 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:57.689862 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:57.690467 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:57.690472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:57.690779 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:57.690861 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.690863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.691994 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.691995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.692575 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.692577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.693070 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.693071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.693179 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.693654 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:57.693659 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:57.694919 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:57.696607 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.696614 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.696616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.696813 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.697030 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.697031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.697031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.697095 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.697369 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:57.697376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:57.697426 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 13:40:57.697518 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 13:40:57.697648 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":94,"threads_started":1}
I20260924 13:40:57.697764 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.689608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.25.240.65:35903)
0924 13:40:57.690266 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:57.690270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:57.690280 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:40:57.690320 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:57.690322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:57.690347 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:57.690411 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:57.690954 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.691910 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.692674 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.692979 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.693067 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.695006 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:57.695013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:57.695013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:57.695020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:57.696506 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.696893 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.696896 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.696897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.696955 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.697165 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.697166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.697166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.697292 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:57.697375 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:57.697596 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 13:40:57.697681 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":76,"threads_started":1}
I20260924 13:40:57.698331 10109 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bd8e4446794ee7bd7fade4f1c042eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:37 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:42108
I20260924 13:40:57.698863 10184 tablet_replica.cc:330] stopping tablet replica
I20260924 13:40:57.698976 10184 raft_consensus.cc:2229] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:57.699043 10184 raft_consensus.cc:2258] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:57.699324 10184 ts_tablet_manager.cc:1905] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:57.701537 10184 ts_tablet_manager.cc:1918] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:57.701646 10184 log.cc:1196] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal/wals/87bd8e4446794ee7bd7fade4f1c042eb
I20260924 13:40:57.701924 10184 ts_tablet_manager.cc:1939] T 87bd8e4446794ee7bd7fade4f1c042eb P 8f6fdd93a38840ada721cefaaa3c64de: Deleting consensus metadata
I20260924 13:40:57.702709 9971 catalog_manager.cc:4841] TS 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903): tablet 87bd8e4446794ee7bd7fade4f1c042eb (table test-table [id=ae2a82786c974610af99528009a7b551]) successfully deleted
W20260924 13:40:57.769335 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:57.769526 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:57.769547 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:57.770630 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:57.770663 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:57.770692 10179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:40:57.772168 10179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:46009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=33409
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:57.772351 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:57.772512 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:57.777629 10192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:57.777676 10179 server_base.cc:1034] running on GCE node
W20260924 13:40:57.777638 10190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:57.777647 10189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:57.778069 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:57.778332 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:57.779491 10179 hybrid_clock.cc:648] HybridClock initialized: now 1790257257779478 us; error 26 us; skew 500 ppm
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:57.781981 10179 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:40:57.782332 10179 webserver.cc:460] Webserver started at http://127.25.240.66:33409/ using document root <none> and password file <none>
I20260924 13:40:57.782507 10179 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:57.782557 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:57.783753 10179 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:57.784399 10199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:57.784585 10179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:40:57.784653 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "cb198887250c4e85a2ce5b788da38c36"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "3b6613221a98dfd6ad1dce07359c3134"
server_key_iv: "3d02673c8a45632acb3e8cbdfcee5f90"
server_key_version: "encryptionkey@0"
I20260924 13:40:57.784957 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:57.797703 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:57.799954 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:57.800063 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:57.800237 10179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:57.800666 10206 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:40:57.802268 10179 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:40:57.802310 10179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:40:57.802335 10179 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:40:57.802942 10179 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:40:57.802974 10179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 13:40:57.803083 10206 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Bootstrap starting.
I20260924 13:40:57.810036 10179 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:46009
I20260924 13:40:57.810138 10313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:46009 every 8 connection(s)
I20260924 13:40:57.810429 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:40:57.814502 10206 log.cc:824] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:57.814895 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 10179
I20260924 13:40:57.815173 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:34585
--local_ip_for_outbound_sockets=127.25.240.67
--tserver_master_addrs=127.25.240.126:39185
--webserver_port=35523
--webserver_interface=127.25.240.67
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:40:57.817272 10206 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:40:57.817668 10206 tablet_bootstrap.cc:492] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Bootstrap complete.
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:57.818828 10206 ts_tablet_manager.cc:1397] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20260924 13:40:57.820480 10206 raft_consensus.cc:348] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:57.820818 10206 raft_consensus.cc:729] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb198887250c4e85a2ce5b788da38c36, State: Initialized, Role: FOLLOWER
I20260924 13:40:57.820982 10206 consensus_queue.cc:260] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } }
I20260924 13:40:57.821309 10206 ts_tablet_manager.cc:1428] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260924 13:40:57.821815 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.811205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:54349 (local address 127.25.240.126:39185)
0924 13:40:57.811282 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:57.811284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:57.811502 (+ 218us) server_negotiation.cc:407] Connection header received
0924 13:40:57.812358 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:57.812358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:57.812370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:57.812413 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:57.813453 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.814003 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.815628 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.815965 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.816035 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.818591 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:57.818593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:57.818593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:57.818602 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:57.820277 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.820722 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.820723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.820723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.820791 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.821099 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.821100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.821100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.821263 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:57.821343 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:57.821659 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 13:40:57.821745 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 13:40:57.822048 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.811143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:54349)
0924 13:40:57.811341 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:57.811393 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:57.812237 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:57.812490 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:57.812496 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:57.813007 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:57.813342 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.813350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.814123 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.814124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.815514 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.815517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.816081 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.816082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.816228 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.816809 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:57.816820 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:57.818501 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:57.820372 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.820377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.820379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.820637 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.820873 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.820874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.820874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.821018 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.821333 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:57.821337 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:57.821497 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 13:40:57.821631 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 13:40:57.821807 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
I20260924 13:40:57.822641 10314 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39185
I20260924 13:40:57.822763 10314 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:57.822935 10314 heartbeater.cc:511] Master 127.25.240.126:39185 requested a full tablet report, sending...
I20260924 13:40:57.823520 9987 ts_manager.cc:194] Registered new tserver with Master: cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009)
I20260924 13:40:57.824203 9987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:54349
I20260924 13:40:57.825315 10314 heartbeater.cc:503] Master 127.25.240.126:39185 was elected leader, sending a full tablet report...
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.66@KRBTEST.COM
I20260924 13:40:57.832113 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.824052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41498 (local address 127.25.240.66:46009)
0924 13:40:57.824216 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:57.824221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:57.824232 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:40:57.824327 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:57.824329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:57.824346 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:57.824460 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:57.824769 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.826039 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.826799 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.827290 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.827374 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.829150 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:57.829157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:57.829157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:57.829165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:57.831034 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.831364 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.831366 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.831367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.831424 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.831590 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.831591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.831591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.831742 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:57.831815 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:57.831915 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 13:40:57.832020 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260924 13:40:57.832129 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.823866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46009 (local address 127.0.0.1:41498)
0924 13:40:57.823925 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:57.823937 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:57.823973 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:57.824466 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:57.824467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:57.824632 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:57.824691 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.824693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.826146 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.826147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.826715 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.826717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.827401 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.827402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.827447 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.827859 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:57.827864 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:57.829065 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:57.831113 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.831114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.831114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.831280 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.831485 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.831485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.831485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.831536 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.831801 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:57.831802 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:57.831834 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 13:40:57.831883 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:40:57.832020 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 13:40:57.832677 10248 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bd8e4446794ee7bd7fade4f1c042eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:37 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:41498
I20260924 13:40:57.833127 10321 tablet_replica.cc:330] stopping tablet replica
I20260924 13:40:57.833250 10321 raft_consensus.cc:2229] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:57.833338 10321 raft_consensus.cc:2258] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:57.833655 10321 ts_tablet_manager.cc:1905] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:57.835988 10321 ts_tablet_manager.cc:1918] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:57.836047 10321 log.cc:1196] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/87bd8e4446794ee7bd7fade4f1c042eb
I20260924 13:40:57.836282 10321 ts_tablet_manager.cc:1939] T 87bd8e4446794ee7bd7fade4f1c042eb P cb198887250c4e85a2ce5b788da38c36: Deleting consensus metadata
I20260924 13:40:57.836726 9972 catalog_manager.cc:4841] TS cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009): tablet 87bd8e4446794ee7bd7fade4f1c042eb (table test-table [id=ae2a82786c974610af99528009a7b551]) successfully deleted
W20260924 13:40:57.901389 10318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:40:57.901599 10318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:40:57.901624 10318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:40:57.902693 10318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:40:57.902726 10318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:40:57.902751 10318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:40:57.904153 10318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:34585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=35523
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:40:57.904336 10318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:40:57.904500 10318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:40:57.909806 10326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:40:57.909823 10329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:57.909821 10318 server_base.cc:1034] running on GCE node
W20260924 13:40:57.909823 10327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:40:57.910230 10318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:40:57.910490 10318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:40:57.911696 10318 hybrid_clock.cc:648] HybridClock initialized: now 1790257257911685 us; error 42 us; skew 500 ppm
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:40:57.914183 10318 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:40:57.914561 10318 webserver.cc:460] Webserver started at http://127.25.240.67:35523/ using document root <none> and password file <none>
I20260924 13:40:57.914754 10318 fs_manager.cc:359] Metadata directory not provided
I20260924 13:40:57.914805 10318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:40:57.916018 10318 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:57.916554 10336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:57.916718 10318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:57.916797 10318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "20a275cf02194c7780533e64d5a4ff98"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "c58ea469ef6bdc817acebc176ce24486"
server_key_iv: "75d89281f9e59170381aabd8d18e3c04"
server_key_version: "encryptionkey@0"
I20260924 13:40:57.917089 10318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:40:57.929842 10318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:40:57.932281 10318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:40:57.932399 10318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:40:57.932564 10318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:40:57.932796 10318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:40:57.932832 10318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:57.932853 10318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:40:57.932866 10318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:40:57.938470 10318 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:34585
I20260924 13:40:57.938539 10449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:34585 every 8 connection(s)
I20260924 13:40:57.938792 10318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:40:57.944159 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 10318
Sep 24 13:40:57 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257257, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:40:57.948454 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.939477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:40271 (local address 127.25.240.126:39185)
0924 13:40:57.939546 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:57.939548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:57.939720 (+ 172us) server_negotiation.cc:407] Connection header received
0924 13:40:57.940196 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:57.940197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:57.940207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:57.940258 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:57.941123 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.941652 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.942569 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.942748 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.942797 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.945228 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:57.945229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:57.945229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:57.945235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:57.946834 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.947281 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.947283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.947283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.947346 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.947641 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.947642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.947642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.947775 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:57.947864 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:57.948261 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 13:40:57.948347 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:40:57.948549 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.939450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:40271)
0924 13:40:57.939602 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:57.939643 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:57.940116 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:57.940319 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:57.940322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:57.940862 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:40:57.941052 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.941056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.941730 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.941731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.942502 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.942504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.942829 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.942830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.942927 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.943376 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:40:57.943385 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:40:57.945118 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:40:57.946921 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.946928 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.946930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.947156 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.947409 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:40:57.947410 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:40:57.947411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:40:57.947555 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:40:57.947912 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:40:57.947920 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:40:57.948089 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 13:40:57.948225 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 13:40:57.948356 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260924 13:40:57.949162 10450 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39185
I20260924 13:40:57.949268 10450 heartbeater.cc:464] Registering TS with master...
I20260924 13:40:57.949463 10450 heartbeater.cc:511] Master 127.25.240.126:39185 requested a full tablet report, sending...
I20260924 13:40:57.949900 9987 ts_manager.cc:194] Registered new tserver with Master: 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585)
I20260924 13:40:57.950275 9987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:40271
I20260924 13:40:57.951773 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.944569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.25.240.126:39185)
0924 13:40:57.944769 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:57.944773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:57.944790 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:40:57.944813 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:57.944813 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:57.944823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:57.944887 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:57.945575 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.946289 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.947260 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.947566 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.947639 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.949098 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:57.949100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:57.949101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:57.949107 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:57.950671 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.951025 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.951026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.951027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.951085 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:57.951282 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:57.951283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:57.951283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:57.951388 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:57.951556 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:57.951597 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 13:40:57.951677 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260924 13:40:57.952517 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:40:57.955078 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.952844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.25.240.126:39185)
0924 13:40:57.952881 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:57.952883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:57.953028 (+ 145us) server_negotiation.cc:407] Connection header received
0924 13:40:57.953069 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:57.953070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:57.953093 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:57.953152 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:40:57.953221 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.953722 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.954459 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.954634 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.954691 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.954820 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:40:57.954899 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:40:57.954952 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:57.954984 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 13:40:57.955000 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:40:57.955367 9987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:59490:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:40:57.956899 9987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:40:57.958830 10109 tablet_service.cc:1467] Processing CreateTablet for tablet dec413a2ec974b38a6ea76fd308a0774 (DEFAULT_TABLE table=test-table [id=be3706ff79124833853cf2284c758c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:57.959023 10109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec413a2ec974b38a6ea76fd308a0774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:57.959111 10248 tablet_service.cc:1467] Processing CreateTablet for tablet dec413a2ec974b38a6ea76fd308a0774 (DEFAULT_TABLE table=test-table [id=be3706ff79124833853cf2284c758c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:57.959318 10248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec413a2ec974b38a6ea76fd308a0774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:57.960596 10067 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Bootstrap starting.
I20260924 13:40:57.960839 10206 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: Bootstrap starting.
I20260924 13:40:57.961050 10067 tablet_bootstrap.cc:654] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:57.961120 10206 tablet_bootstrap.cc:654] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:57.961841 10206 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: No bootstrap required, opened a new log
I20260924 13:40:57.961890 10206 ts_tablet_manager.cc:1397] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:40:57.961998 10067 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: No bootstrap required, opened a new log
I20260924 13:40:57.962031 10067 ts_tablet_manager.cc:1397] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:57.962237 10206 raft_consensus.cc:348] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:57.962313 10206 raft_consensus.cc:374] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:57.962352 10206 raft_consensus.cc:729] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb198887250c4e85a2ce5b788da38c36, State: Initialized, Role: FOLLOWER
I20260924 13:40:57.962332 10067 raft_consensus.cc:348] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:57.962397 10067 raft_consensus.cc:374] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:57.962409 10206 consensus_queue.cc:260] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:57.962445 10067 raft_consensus.cc:729] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6fdd93a38840ada721cefaaa3c64de, State: Initialized, Role: FOLLOWER
I20260924 13:40:57.962494 10206 ts_tablet_manager.cc:1428] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 13:40:57.962493 10067 consensus_queue.cc:260] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:57.962582 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.958665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34585 (local address 127.0.0.1:42814)
0924 13:40:57.958710 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:57.958720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:57.958784 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:57.959192 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:57.959193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:57.959195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:57.959232 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.959234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.960503 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.960504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.961286 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:57.961288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.962303 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.962304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:57.962371 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.962392 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 13:40:57.962454 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 13:40:57.962493 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 13:40:57.962603 10067 ts_tablet_manager.cc:1428] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 13:40:57.962990 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:57.958732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.25.240.67:34585)
0924 13:40:57.958965 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:57.958970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:57.958981 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:40:57.959021 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:57.959023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:57.959057 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:57.959168 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:57.959327 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.960414 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.961406 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:57.962205 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:57.962284 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:57.962354 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:57.962801 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 13:40:57.962852 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260924 13:40:57.963613 10384 tablet_service.cc:1467] Processing CreateTablet for tablet dec413a2ec974b38a6ea76fd308a0774 (DEFAULT_TABLE table=test-table [id=be3706ff79124833853cf2284c758c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:40:57.963897 10384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dec413a2ec974b38a6ea76fd308a0774. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:40:57.966498 10459 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: Bootstrap starting.
I20260924 13:40:57.967198 10459 tablet_bootstrap.cc:654] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: Neither blocks nor log segments found. Creating new log.
I20260924 13:40:57.967628 10459 log.cc:824] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: Log is configured to *not* fsync() on all Append() calls
I20260924 13:40:57.968240 10459 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: No bootstrap required, opened a new log
I20260924 13:40:57.968295 10459 ts_tablet_manager.cc:1397] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:40:57.969542 10459 raft_consensus.cc:348] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:57.969777 10459 raft_consensus.cc:374] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:40:57.969802 10459 raft_consensus.cc:729] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a275cf02194c7780533e64d5a4ff98, State: Initialized, Role: FOLLOWER
I20260924 13:40:57.969913 10459 consensus_queue.cc:260] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:57.970185 10459 ts_tablet_manager.cc:1428] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:40:57.970250 10450 heartbeater.cc:503] Master 127.25.240.126:39185 was elected leader, sending a full tablet report...
I20260924 13:40:58.048940 10319 raft_consensus.cc:482] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:40:58.049073 10319 raft_consensus.cc:504] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:58.049491 10319 leader_election.cc:288] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903)
I20260924 13:40:58.052661 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:58.049691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34585 (local address 127.25.240.66:34507)
0924 13:40:58.049751 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:58.049765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:58.049857 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:58.050008 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:58.050009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:58.050011 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:58.050095 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:58.050097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.050792 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.050793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:58.051709 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:58.051711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.052355 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.052356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:58.052433 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:58.052461 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 13:40:58.052517 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 13:40:58.052560 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:40:58.052663 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:58.049804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:34507 (local address 127.25.240.67:34585)
0924 13:40:58.049857 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:58.049859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:58.049869 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:40:58.049945 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:58.049946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:58.049956 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:58.050006 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:58.050178 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.050715 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.051794 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.052280 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.052338 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:58.052345 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:58.052531 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 13:40:58.052573 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 13:40:58.052791 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:58.049691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:35903 (local address 127.25.240.66:59619)
0924 13:40:58.049898 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:58.049908 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:40:58.049969 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:40:58.050213 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:40:58.050214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:40:58.050215 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:40:58.050280 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:58.050281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.051018 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.051019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:58.051809 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:40:58.051811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.052509 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.052510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:40:58.052578 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:58.052582 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:40:58.052643 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 13:40:58.052685 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":49,"thread_start_us":99,"threads_started":1}
I20260924 13:40:58.052789 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:58.049792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:59619 (local address 127.25.240.65:35903)
0924 13:40:58.049856 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:58.049858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:58.049961 (+ 103us) server_negotiation.cc:407] Connection header received
0924 13:40:58.050055 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:58.050056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:58.050084 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:58.050123 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:40:58.050367 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.050937 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.051888 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.052397 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.052471 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:58.052574 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:58.052652 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 13:40:58.052700 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:40:58.053009 10404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec413a2ec974b38a6ea76fd308a0774" candidate_uuid: "cb198887250c4e85a2ce5b788da38c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a275cf02194c7780533e64d5a4ff98" is_pre_election: true
I20260924 13:40:58.053094 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec413a2ec974b38a6ea76fd308a0774" candidate_uuid: "cb198887250c4e85a2ce5b788da38c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6fdd93a38840ada721cefaaa3c64de" is_pre_election: true
I20260924 13:40:58.053176 10404 raft_consensus.cc:2454] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb198887250c4e85a2ce5b788da38c36 in term 0.
I20260924 13:40:58.053249 10129 raft_consensus.cc:2454] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb198887250c4e85a2ce5b788da38c36 in term 0.
I20260924 13:40:58.053427 10202 leader_election.cc:302] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a275cf02194c7780533e64d5a4ff98, cb198887250c4e85a2ce5b788da38c36; no voters:
I20260924 13:40:58.053575 10319 raft_consensus.cc:2781] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:40:58.053644 10319 raft_consensus.cc:482] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:40:58.053687 10319 raft_consensus.cc:3037] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:58.054381 10319 raft_consensus.cc:504] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:58.054527 10319 leader_election.cc:288] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [CANDIDATE]: Term 1 election: Requested vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903)
I20260924 13:40:58.054708 10404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec413a2ec974b38a6ea76fd308a0774" candidate_uuid: "cb198887250c4e85a2ce5b788da38c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a275cf02194c7780533e64d5a4ff98"
I20260924 13:40:58.054734 10129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dec413a2ec974b38a6ea76fd308a0774" candidate_uuid: "cb198887250c4e85a2ce5b788da38c36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f6fdd93a38840ada721cefaaa3c64de"
I20260924 13:40:58.054811 10129 raft_consensus.cc:3037] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:58.054817 10404 raft_consensus.cc:3037] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:40:58.055490 10129 raft_consensus.cc:2454] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb198887250c4e85a2ce5b788da38c36 in term 1.
I20260924 13:40:58.055583 10404 raft_consensus.cc:2454] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb198887250c4e85a2ce5b788da38c36 in term 1.
I20260924 13:40:58.055657 10200 leader_election.cc:302] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f6fdd93a38840ada721cefaaa3c64de, cb198887250c4e85a2ce5b788da38c36; no voters:
I20260924 13:40:58.055747 10319 raft_consensus.cc:2781] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:40:58.055888 10319 raft_consensus.cc:686] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 1 LEADER]: Becoming Leader. State: Replica: cb198887250c4e85a2ce5b788da38c36, State: Running, Role: LEADER
I20260924 13:40:58.055971 10319 consensus_queue.cc:237] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:40:58.056692 9987 catalog_manager.cc:5495] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb198887250c4e85a2ce5b788da38c36 (127.25.240.66). New cstate: current_term: 1 leader_uuid: "cb198887250c4e85a2ce5b788da38c36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } health_report { overall_health: HEALTHY } } }
W20260924 13:40:58.061008 10315 tablet.cc:2369] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:40:58.077646 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:58.075245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.25.240.66:46009)
0924 13:40:58.075320 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:58.075322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:58.075333 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:40:58.075405 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:58.075405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:58.075423 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:58.075503 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:40:58.075631 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.076132 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.076869 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.077078 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.077153 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:58.077334 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:40:58.077380 (+ 46us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 13:40:58.077525 (+ 145us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41510: token expired
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 13:40:58.077713 10320 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41510: token expired
I20260924 13:40:58.083403 10454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:58.078023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.25.240.126:39185)
0924 13:40:58.078086 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:58.078087 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:58.078095 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:40:58.078122 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:58.078122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:58.078130 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:58.078173 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:40:58.078668 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.079105 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.079750 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.079925 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.079971 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:58.081074 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:40:58.081075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:40:58.081075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:40:58.081082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:40:58.082453 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:58.082801 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:58.082802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:58.082802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:58.082850 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:40:58.083018 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:40:58.083018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:40:58.083019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:40:58.083113 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:40:58.083217 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:58.083281 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 13:40:58.083344 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:40:58.086308 10320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:58.084380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.25.240.66:46009)
0924 13:40:58.084436 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:40:58.084437 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:40:58.084446 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:40:58.084466 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:40:58.084466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:40:58.084474 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:40:58.084514 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:40:58.084582 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.085020 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.085682 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:40:58.085858 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:40:58.085913 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:40:58.086009 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:40:58.086090 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:40:58.086127 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 13:40:58.086178 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 13:40:58.086192 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:40:58.087841 10404 raft_consensus.cc:1261] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 1 FOLLOWER]: Refusing update from remote peer cb198887250c4e85a2ce5b788da38c36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:58.087857 10129 raft_consensus.cc:1261] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Refusing update from remote peer cb198887250c4e85a2ce5b788da38c36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:40:58.088214 10463 consensus_queue.cc:1035] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:58.088349 10319 consensus_queue.cc:1035] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:40:58.090692 10465 mvcc.cc:205] Tried to move back new op lower bound from 7332893729125212160 to 7332893728997765120. Current Snapshot: MvccSnapshot[applied={T|T < 7332893729125212160}]
I20260924 13:40:58.091056 10466 mvcc.cc:205] Tried to move back new op lower bound from 7332893729125212160 to 7332893728997765120. Current Snapshot: MvccSnapshot[applied={T|T < 7332893729125212160}]
I20260924 13:40:58.091706 10467 mvcc.cc:205] Tried to move back new op lower bound from 7332893729125212160 to 7332893728997765120. Current Snapshot: MvccSnapshot[applied={T|T < 7332893729125212160}]
I20260924 13:40:58.092871 9987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:58.092964 9987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:40:58.093729 9987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Sending DeleteTablet for 3 replicas of tablet dec413a2ec974b38a6ea76fd308a0774
I20260924 13:40:58.094009 10248 tablet_service.cc:1514] Processing DeleteTablet for tablet dec413a2ec974b38a6ea76fd308a0774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:58 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:41498
I20260924 13:40:58.094048 10109 tablet_service.cc:1514] Processing DeleteTablet for tablet dec413a2ec974b38a6ea76fd308a0774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:58 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:42108
I20260924 13:40:58.094094 10321 tablet_replica.cc:330] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: stopping tablet replica
I20260924 13:40:58.094141 10321 raft_consensus.cc:2229] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:40:58.094215 10321 raft_consensus.cc:2258] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:58.094177 10384 tablet_service.cc:1514] Processing DeleteTablet for tablet dec413a2ec974b38a6ea76fd308a0774 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:40:58 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:42814
I20260924 13:40:58.094295 10184 tablet_replica.cc:330] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: stopping tablet replica
I20260924 13:40:58.094362 10184 raft_consensus.cc:2229] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:58.094427 10184 raft_consensus.cc:2258] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:58.094419 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 10025
I20260924 13:40:58.094517 10476 tablet_replica.cc:330] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: stopping tablet replica
I20260924 13:40:58.094640 10476 raft_consensus.cc:2229] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:40:58.094798 10476 raft_consensus.cc:2258] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:40:58.095170 10321 ts_tablet_manager.cc:1905] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:58.096553 10476 ts_tablet_manager.cc:1905] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:40:58.096611 10321 ts_tablet_manager.cc:1918] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:58.096669 10321 log.cc:1196] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/dec413a2ec974b38a6ea76fd308a0774
I20260924 13:40:58.096889 10321 ts_tablet_manager.cc:1939] T dec413a2ec974b38a6ea76fd308a0774 P cb198887250c4e85a2ce5b788da38c36: Deleting consensus metadata
I20260924 13:40:58.097187 9972 catalog_manager.cc:4841] TS cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009): tablet dec413a2ec974b38a6ea76fd308a0774 (table test-table [id=be3706ff79124833853cf2284c758c35]) successfully deleted
I20260924 13:40:58.098186 10476 ts_tablet_manager.cc:1918] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:40:58.098271 10476 log.cc:1196] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/dec413a2ec974b38a6ea76fd308a0774
I20260924 13:40:58.098520 10476 ts_tablet_manager.cc:1939] T dec413a2ec974b38a6ea76fd308a0774 P 20a275cf02194c7780533e64d5a4ff98: Deleting consensus metadata
I20260924 13:40:58.099272 9973 catalog_manager.cc:4841] TS 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585): tablet dec413a2ec974b38a6ea76fd308a0774 (table test-table [id=be3706ff79124833853cf2284c758c35]) successfully deleted
W20260924 13:40:58.102001 9971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:35903 (error 108)
I20260924 13:40:58.102267 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 10179
I20260924 13:40:58.102862 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:40:58.102347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:35903 (local address 127.0.0.1:42120)
0924 13:40:58.102780 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 13:40:58.102838 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415}
W20260924 13:40:58.103085 9971 catalog_manager.cc:4584] TS 8f6fdd93a38840ada721cefaaa3c64de (127.25.240.65:35903): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dec413a2ec974b38a6ea76fd308a0774: Network error: Client connection negotiation failed: client connection to 127.25.240.65:35903: connect: Connection refused (error 111)
I20260924 13:40:58.107168 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 10318
I20260924 13:40:58.111522 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 9954
I20260924 13:41:18.116354 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39185
--webserver_interface=127.25.240.126
--webserver_port=39027
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:39185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:18.197773 10478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:18.197979 10478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:18.198001 10478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:18.199203 10478 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 13:41:18.199244 10478 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 13:41:18.199256 10478 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 13:41:18.199267 10478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:18.199280 10478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:18.199293 10478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:18.199306 10478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:18.199358 10478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:18.200810 10478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:39185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=39027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:18.201064 10478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:18.201303 10478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:41:18.206426 10478 server_base.cc:1034] running on GCE node
W20260924 13:41:18.206382 10483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:18.206365 10486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:18.206396 10484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:18.206841 10478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:18.207134 10478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:18.208273 10478 hybrid_clock.cc:648] HybridClock initialized: now 1790257278208251 us; error 37 us; skew 500 ppm
Sep 24 13:41:18 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257278, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:18.210711 10478 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:41:18.211184 10478 webserver.cc:460] Webserver started at http://127.25.240.126:39027/ using document root <none> and password file <none>
I20260924 13:41:18.211418 10478 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:18.211495 10478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:18.212837 10478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:18.213502 10493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:18.213702 10478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:18.213800 10478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "696fa76b72764e84bc9fab5f125af409"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "922e10ae863f1478471ef50780d4801c"
server_key_iv: "499940cb8529bad45b8e7e60195f47d1"
server_key_version: "encryptionkey@0"
I20260924 13:41:18.214134 10478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:18.225664 10478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:18.227922 10478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:18.228088 10478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:18.232291 10478 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:39185
I20260924 13:41:18.232323 10545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:39185 every 8 connection(s)
I20260924 13:41:18.232690 10478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:18.234119 10546 sys_catalog.cc:263] Verifying existing consensus state
I20260924 13:41:18.234798 10546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Bootstrap starting.
I20260924 13:41:18.241771 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 10478
I20260924 13:41:18.242051 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:35903
--local_ip_for_outbound_sockets=127.25.240.65
--tserver_master_addrs=127.25.240.126:39185
--webserver_port=40501
--webserver_interface=127.25.240.65
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:41:18.243618 10546 log.cc:824] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:18.247092 10546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:41:18.247457 10546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Bootstrap complete.
I20260924 13:41:18.251405 10546 raft_consensus.cc:348] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:41:18.251818 10546 raft_consensus.cc:729] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 696fa76b72764e84bc9fab5f125af409, State: Initialized, Role: FOLLOWER
I20260924 13:41:18.251997 10546 consensus_queue.cc:260] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:41:18.252096 10546 raft_consensus.cc:388] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:18.252142 10546 raft_consensus.cc:482] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:18.252202 10546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 2 FOLLOWER]: Advancing to term 3
I20260924 13:41:18.253793 10546 raft_consensus.cc:504] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:41:18.253907 10546 leader_election.cc:302] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 696fa76b72764e84bc9fab5f125af409; no voters:
I20260924 13:41:18.254056 10546 leader_election.cc:288] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 13:41:18.254136 10550 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 13:41:18.254292 10550 raft_consensus.cc:686] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [term 3 LEADER]: Becoming Leader. State: Replica: 696fa76b72764e84bc9fab5f125af409, State: Running, Role: LEADER
I20260924 13:41:18.254297 10546 sys_catalog.cc:564] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:18.254442 10550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } }
I20260924 13:41:18.254711 10551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "696fa76b72764e84bc9fab5f125af409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } } }
I20260924 13:41:18.254725 10552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 696fa76b72764e84bc9fab5f125af409. Latest consensus state: current_term: 3 leader_uuid: "696fa76b72764e84bc9fab5f125af409" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "696fa76b72764e84bc9fab5f125af409" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39185 } } }
I20260924 13:41:18.254822 10551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:18.254825 10552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:18.255074 10558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:18.255867 10558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:18.255993 10558 catalog_manager.cc:1259] Loaded cluster ID: 86590024d35b481fbf96c4cf25550d33
I20260924 13:41:18.256048 10558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:18.257400 10558 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:18.257596 10558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Loaded TSK: 0
I20260924 13:41:18.257647 10558 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 696fa76b72764e84bc9fab5f125af409: Loaded TSK: 1
F20260924 13:41:18.257874 10558 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbc189aa9ad google::LogMessage::Fail() at ??:0
@ 0x7fbc189ad88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbc189aa4de google::LogMessage::Flush() at ??:0
@ 0x7fbc189aba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbc1b117104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbc1b114d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbc209041cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbc2090448e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbc20904594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbc208cac48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fbc208d4ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbc1a8780d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbc1a86d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbc1b6d16db start_thread at ??:0
@ 0x7fbc174a861f clone at ??:0
W20260924 13:41:18.337031 10548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:18.337284 10548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:18.337324 10548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:18.338553 10548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:18.338624 10548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:18.338670 10548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:41:18.340161 10548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:35903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=40501
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:18.340433 10548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:18.340672 10548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:18.345731 10588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:18.345749 10548 server_base.cc:1034] running on GCE node
W20260924 13:41:18.345853 10587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:18.345728 10590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:18.346091 10548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:18.346385 10548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:18.347545 10548 hybrid_clock.cc:648] HybridClock initialized: now 1790257278347528 us; error 34 us; skew 500 ppm
Sep 24 13:41:18 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257278, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:18.350286 10548 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:41:18.350725 10548 webserver.cc:460] Webserver started at http://127.25.240.65:40501/ using document root <none> and password file <none>
I20260924 13:41:18.350983 10548 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:18.351056 10548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:18.352423 10548 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:18.353160 10597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:18.353374 10548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:18.353471 10548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "8f6fdd93a38840ada721cefaaa3c64de"
format_stamp: "Formatted at 2026-09-24 13:40:36 on dist-test-slave-m5z1"
server_key: "ce56f7d356a7d0643094dfd1ecf4c404"
server_key_iv: "33b1be60ac7d78bfa3aada3069e743f3"
server_key_version: "encryptionkey@0"
I20260924 13:41:18.353868 10548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:18.364585 10548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:18.367120 10548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:18.367318 10548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:18.367578 10548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:18.368073 10604 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 13:41:18.370116 10548 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 13:41:18.370190 10548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 13:41:18.370235 10548 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 13:41:18.370959 10548 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 13:41:18.371023 10548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 13:41:18.371088 10604 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Bootstrap starting.
I20260924 13:41:18.379259 10548 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:35903
I20260924 13:41:18.379678 10548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:41:18.381760 10711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:35903 every 8 connection(s)
I20260924 13:41:18.382797 10604 log.cc:824] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:18.385538 10604 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 13:41:18.385878 10604 tablet_bootstrap.cc:492] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Bootstrap complete.
I20260924 13:41:18.387125 10604 ts_tablet_manager.cc:1397] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.002s
I20260924 13:41:18.387933 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 10548
I20260924 13:41:18.388224 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:46009
--local_ip_for_outbound_sockets=127.25.240.66
--tserver_master_addrs=127.25.240.126:39185
--webserver_port=33409
--webserver_interface=127.25.240.66
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:41:18.388636 10604 raft_consensus.cc:348] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:41:18.388964 10604 raft_consensus.cc:729] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f6fdd93a38840ada721cefaaa3c64de, State: Initialized, Role: FOLLOWER
I20260924 13:41:18.389120 10604 consensus_queue.cc:260] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:41:18.389392 10604 ts_tablet_manager.cc:1428] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
W20260924 13:41:18.458845 10714 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:41:18.380403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:48909)
0924 13:41:18.380593 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.380641 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:18.381117 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:18.458645 (+ 77528us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":111,"threads_started":1}
W20260924 13:41:18.459125 10712 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:39185 (0 consecutive failures): Network error: Failed to ping master at 127.25.240.126:39185: Client connection negotiation failed: client connection to 127.25.240.126:39185: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 13:41:18.459519 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:18.459383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:59429)
0924 13:41:18.459443 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.459488 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:41:18.459901 10714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:18.459802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:45191)
0924 13:41:18.459851 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.459879 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 13:41:18.460045 10712 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 13:41:18.472141 10717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:18.472328 10717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:18.472350 10717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:18.473505 10717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:18.473546 10717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:18.473639 10717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:41:18.475082 10717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:46009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=33409
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:18.475317 10717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:18.475536 10717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:18.480461 10723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:18.480455 10722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:18.480680 10717 server_base.cc:1034] running on GCE node
W20260924 13:41:18.480456 10725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:18.481058 10717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:18.481258 10717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:18.482412 10717 hybrid_clock.cc:648] HybridClock initialized: now 1790257278482389 us; error 38 us; skew 500 ppm
Sep 24 13:41:18 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257278, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:18.484834 10717 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:41:18.485261 10717 webserver.cc:460] Webserver started at http://127.25.240.66:33409/ using document root <none> and password file <none>
I20260924 13:41:18.485500 10717 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:18.485546 10717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:18.486899 10717 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:18.487442 10732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:18.487612 10717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:18.487687 10717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "cb198887250c4e85a2ce5b788da38c36"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "3b6613221a98dfd6ad1dce07359c3134"
server_key_iv: "3d02673c8a45632acb3e8cbdfcee5f90"
server_key_version: "encryptionkey@0"
I20260924 13:41:18.487990 10717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:18.509943 10717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:18.512202 10717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:18.512313 10717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:18.512497 10717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:18.512742 10717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:18.512773 10717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:18.512794 10717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:18.512805 10717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:18.518555 10717 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:46009
I20260924 13:41:18.518639 10845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:46009 every 8 connection(s)
I20260924 13:41:18.518975 10717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:41:18.520072 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:18.519633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:39279)
0924 13:41:18.519848 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.519935 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
W20260924 13:41:18.520469 10846 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:39185 (0 consecutive failures): Network error: Failed to ping master at 127.25.240.126:39185: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
I20260924 13:41:18.520766 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:18.520663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:45307)
0924 13:41:18.520720 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.520747 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 13:41:18.521160 10848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:18.521084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:56773)
0924 13:41:18.521123 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.521145 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:41:18.521293 10846 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:41:18.523612 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 10717
I20260924 13:41:18.523871 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:34585
--local_ip_for_outbound_sockets=127.25.240.67
--tserver_master_addrs=127.25.240.126:39185
--webserver_port=35523
--webserver_interface=127.25.240.67
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:41751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:18.603426 10849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:18.603644 10849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:18.603667 10849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:18.604810 10849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:18.604854 10849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:18.604880 10849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:41:18.606370 10849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:34585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=35523
--tserver_master_addrs=127.25.240.126:39185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:18.606616 10849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:18.606837 10849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:18.611759 10854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:18.611718 10855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:18.611961 10849 server_base.cc:1034] running on GCE node
W20260924 13:41:18.612105 10857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:18.612325 10849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:18.612562 10849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:18.613709 10849 hybrid_clock.cc:648] HybridClock initialized: now 1790257278613704 us; error 52 us; skew 500 ppm
Sep 24 13:41:18 dist-test-slave-m5z1 krb5kdc[9406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257278, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:18.616175 10849 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:41:18.616616 10849 webserver.cc:460] Webserver started at http://127.25.240.67:35523/ using document root <none> and password file <none>
I20260924 13:41:18.616798 10849 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:18.616837 10849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:18.618121 10849 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:18.618690 10864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:18.618913 10849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:41:18.619006 10849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "20a275cf02194c7780533e64d5a4ff98"
format_stamp: "Formatted at 2026-09-24 13:40:37 on dist-test-slave-m5z1"
server_key: "c58ea469ef6bdc817acebc176ce24486"
server_key_iv: "75d89281f9e59170381aabd8d18e3c04"
server_key_version: "encryptionkey@0"
I20260924 13:41:18.619326 10849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:18.635241 10849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:18.637004 10849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:18.637156 10849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:18.637382 10849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:18.637741 10849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:18.637780 10849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:18.637822 10849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:18.637847 10849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:18.643389 10849 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:34585
I20260924 13:41:18.643468 10977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:34585 every 8 connection(s)
I20260924 13:41:18.643793 10849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:41:18.644794 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:18.644437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:56505)
0924 13:41:18.644589 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.644656 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
W20260924 13:41:18.645198 10978 heartbeater.cc:644] Failed to heartbeat to 127.25.240.126:39185 (0 consecutive failures): Network error: Failed to ping master at 127.25.240.126:39185: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
I20260924 13:41:18.645449 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:18.645366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:53409)
0924 13:41:18.645406 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.645432 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 13:41:18.645735 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:18.645663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:40513)
0924 13:41:18.645700 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:18.645721 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 13:41:18.645849 10978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 13:41:18.648873 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 10849
I20260924 13:41:19.460682 10982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:19.460448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:36125)
0924 13:41:19.460580 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:19.460616 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":63,"threads_started":1}
I20260924 13:41:19.521955 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:19.521708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:56603)
0924 13:41:19.521835 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:19.521892 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260924 13:41:19.646304 10984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:19.646119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:40749)
0924 13:41:19.646221 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:19.646255 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260924 13:41:20.056528 10985 raft_consensus.cc:482] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:20.056695 10985 raft_consensus.cc:504] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:41:20.057060 10985 leader_election.cc:288] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009)
Sep 24 13:41:20 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257278, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.67@KRBTEST.COM
Sep 24 13:41:20 dist-test-slave-m5z1 krb5kdc[9406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257278, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.66@KRBTEST.COM
I20260924 13:41:20.067508 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:20.057243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34585 (local address 127.25.240.65:47885)
0924 13:41:20.057357 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:20.057368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:20.057440 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:20.058446 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:20.058447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:20.058836 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:20.059015 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:20.059021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:20.060122 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:20.060123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:20.060904 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:20.060906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:20.061511 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:20.061512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:20.061648 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:20.062243 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:20.062252 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:20.063677 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:20.065819 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:20.065826 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:20.065828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:20.066115 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:20.066350 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:20.066352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:20.066352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:20.066487 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:20.066940 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:20.066946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:20.067095 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 13:41:20.067209 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 13:41:20.067370 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":36,"thread_start_us":52,"threads_started":1}
I20260924 13:41:20.067520 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:20.057393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:47885 (local address 127.25.240.67:34585)
0924 13:41:20.057547 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:20.057594 (+ 47us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:20.057624 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:41:20.058214 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:20.058217 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:20.058288 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:20.058429 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:20.059134 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:20.060001 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:20.060994 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:20.061403 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:20.061530 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:20.063892 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:20.063896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:20.063897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:20.063908 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:20.065740 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:20.066198 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:20.066201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:20.066201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:20.066283 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:20.066560 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:20.066561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:20.066562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:20.066842 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:20.066943 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:20.067254 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 13:41:20.067387 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":53,"threads_started":1}
I20260924 13:41:20.067694 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:20.057253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46009 (local address 127.25.240.65:46745)
0924 13:41:20.057364 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:20.057375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:20.057430 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:20.058445 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:20.058447 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:20.058837 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:20.059015 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:20.059021 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:20.060104 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:20.060105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:20.060902 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:20.060904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:20.061508 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:20.061509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:20.061647 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:20.062297 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:20.062302 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:20.064146 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:20.066446 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:20.066447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:20.066447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:20.066657 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:20.066886 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:20.066887 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:20.066887 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:20.066954 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:20.067380 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:20.067380 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:20.067420 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 13:41:20.067494 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 13:41:20.067592 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":33,"thread_start_us":51,"threads_started":1}
I20260924 13:41:20.068137 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:20.057343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:46745 (local address 127.25.240.66:46009)
0924 13:41:20.057541 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:20.057595 (+ 54us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:20.057617 (+ 22us) server_negotiation.cc:407] Connection header received
0924 13:41:20.058188 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:20.058191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:20.058249 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:20.058403 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:20.059140 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:20.060002 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:20.061004 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:20.061401 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:20.061514 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:20.064277 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:20.064280 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:20.064281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:20.064292 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:20.066369 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:20.066737 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:20.066739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:20.066740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:20.066817 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:20.067016 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:20.067017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:20.067018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:20.067304 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:20.067382 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:20.067905 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 13:41:20.068039 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
W20260924 13:41:20.068418 10600 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
W20260924 13:41:20.068809 10599 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
I20260924 13:41:20.068862 10599 leader_election.cc:302] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f6fdd93a38840ada721cefaaa3c64de; no voters: 20a275cf02194c7780533e64d5a4ff98, cb198887250c4e85a2ce5b788da38c36
I20260924 13:41:20.068969 10985 raft_consensus.cc:2726] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:41:20.461237 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:20.461111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:51663)
0924 13:41:20.461162 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:20.461194 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:41:20.522737 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:20.522523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:42491)
0924 13:41:20.522641 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:20.522690 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
I20260924 13:41:20.646868 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:20.646688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:56315)
0924 13:41:20.646801 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:20.646834 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260924 13:41:21.461925 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:21.461713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:43639)
0924 13:41:21.461833 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:21.461872 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260924 13:41:21.523371 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:21.523178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:53225)
0924 13:41:21.523298 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:21.523335 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260924 13:41:21.647434 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:21.647248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:34219)
0924 13:41:21.647365 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:21.647399 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260924 13:41:21.928813 10995 raft_consensus.cc:482] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:21.928900 10995 raft_consensus.cc:504] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:41:21.929064 10995 leader_election.cc:288] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009)
W20260924 13:41:21.929509 10599 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
W20260924 13:41:21.929610 10600 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
I20260924 13:41:21.929701 10600 leader_election.cc:302] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f6fdd93a38840ada721cefaaa3c64de; no voters: 20a275cf02194c7780533e64d5a4ff98, cb198887250c4e85a2ce5b788da38c36
I20260924 13:41:21.929798 10995 raft_consensus.cc:2726] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:41:22.462575 10996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:22.462355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:37729)
0924 13:41:22.462492 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:22.462527 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260924 13:41:22.523984 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:22.523800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:43209)
0924 13:41:22.523918 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:22.523949 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260924 13:41:22.648007 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:22.647829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:43745)
0924 13:41:22.647944 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:22.647970 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260924 13:41:23.463289 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:23.463074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:33687)
0924 13:41:23.463197 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:23.463241 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260924 13:41:23.524641 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:23.524425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:44293)
0924 13:41:23.524566 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:23.524603 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260924 13:41:23.648669 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:23.648455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:53687)
0924 13:41:23.648594 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:23.648631 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260924 13:41:24.463953 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:24.463719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:56541)
0924 13:41:24.463870 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:24.463907 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
I20260924 13:41:24.525494 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:24.525253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:50557)
0924 13:41:24.525419 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:24.525456 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260924 13:41:24.617057 11004 raft_consensus.cc:482] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:24.617173 11004 raft_consensus.cc:504] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:41:24.617415 11004 leader_election.cc:288] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009)
W20260924 13:41:24.617799 10600 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
W20260924 13:41:24.617890 10599 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
I20260924 13:41:24.617954 10599 leader_election.cc:302] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f6fdd93a38840ada721cefaaa3c64de; no voters: 20a275cf02194c7780533e64d5a4ff98, cb198887250c4e85a2ce5b788da38c36
I20260924 13:41:24.618027 11004 raft_consensus.cc:2726] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:41:24.649300 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:24.649105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:46315)
0924 13:41:24.649226 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:24.649262 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 13:41:25.464601 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:25.464390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:47643)
0924 13:41:25.464515 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:25.464552 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
I20260924 13:41:25.526211 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:25.525982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:57281)
0924 13:41:25.526131 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:25.526169 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
I20260924 13:41:25.650089 11008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:25.649844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:56103)
0924 13:41:25.650007 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:25.650046 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260924 13:41:26.465312 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:26.465043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:41631)
0924 13:41:26.465212 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:26.465255 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":95,"threads_started":1}
I20260924 13:41:26.526856 11010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:26.526623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:34479)
0924 13:41:26.526770 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:26.526816 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260924 13:41:26.650733 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:26.650513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:32993)
0924 13:41:26.650646 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:26.650693 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
I20260924 13:41:27.466059 11012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:27.465813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:45669)
0924 13:41:27.465968 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:27.466011 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260924 13:41:27.527496 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:27.527287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:51643)
0924 13:41:27.527419 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:27.527456 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260924 13:41:27.651398 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:27.651205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:49747)
0924 13:41:27.651320 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:27.651355 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260924 13:41:27.654687 11015 raft_consensus.cc:482] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:27.654850 11015 raft_consensus.cc:504] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:41:27.655054 11015 leader_election.cc:288] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009)
W20260924 13:41:27.655435 10599 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
W20260924 13:41:27.655501 10600 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
I20260924 13:41:27.655581 10600 leader_election.cc:302] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f6fdd93a38840ada721cefaaa3c64de; no voters: 20a275cf02194c7780533e64d5a4ff98, cb198887250c4e85a2ce5b788da38c36
I20260924 13:41:27.655668 11015 raft_consensus.cc:2726] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:41:28.466764 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:28.466572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:41337)
0924 13:41:28.466687 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:28.466719 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260924 13:41:28.528090 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:28.527889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:33733)
0924 13:41:28.528014 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:28.528051 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":53,"threads_started":1}
I20260924 13:41:28.651962 11018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:28.651770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:55235)
0924 13:41:28.651887 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:28.651915 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260924 13:41:29.467509 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:29.467266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:49361)
0924 13:41:29.467421 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:29.467464 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260924 13:41:29.528771 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:29.528573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:46857)
0924 13:41:29.528691 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:29.528730 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260924 13:41:29.652585 11021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:29.652365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:47769)
0924 13:41:29.652511 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:29.652544 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260924 13:41:30.468225 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:30.467991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:49999)
0924 13:41:30.468142 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:30.468177 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260924 13:41:30.529368 11023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:30.529180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:35245)
0924 13:41:30.529295 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:30.529327 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260924 13:41:30.653203 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:30.653004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:37865)
0924 13:41:30.653126 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:30.653163 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260924 13:41:31.468849 11025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:31.468644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:43255)
0924 13:41:31.468766 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:31.468800 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260924 13:41:31.529982 11026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:31.529783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:49047)
0924 13:41:31.529897 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:31.529933 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260924 13:41:31.654048 11027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:31.653838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:42537)
0924 13:41:31.653960 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:31.654002 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260924 13:41:32.279608 11028 raft_consensus.cc:482] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:32.279699 11028 raft_consensus.cc:504] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a275cf02194c7780533e64d5a4ff98" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34585 } } peers { permanent_uuid: "8f6fdd93a38840ada721cefaaa3c64de" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 35903 } } peers { permanent_uuid: "cb198887250c4e85a2ce5b788da38c36" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46009 } }
I20260924 13:41:32.279872 11028 leader_election.cc:288] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585), cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009)
W20260924 13:41:32.280277 10600 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 20a275cf02194c7780533e64d5a4ff98 (127.25.240.67:34585): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
W20260924 13:41:32.280383 10599 leader_election.cc:341] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cb198887250c4e85a2ce5b788da38c36 (127.25.240.66:46009): Not found: Tablet not found: dec413a2ec974b38a6ea76fd308a0774
I20260924 13:41:32.280418 10599 leader_election.cc:302] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f6fdd93a38840ada721cefaaa3c64de; no voters: 20a275cf02194c7780533e64d5a4ff98, cb198887250c4e85a2ce5b788da38c36
I20260924 13:41:32.280503 11028 raft_consensus.cc:2726] T dec413a2ec974b38a6ea76fd308a0774 P 8f6fdd93a38840ada721cefaaa3c64de [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 13:41:32.469547 11029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:32.469324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:42845)
0924 13:41:32.469459 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:32.469498 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260924 13:41:32.530571 11030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:32.530383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:35901)
0924 13:41:32.530498 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:32.530533 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260924 13:41:32.654753 11031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:32.654555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:59275)
0924 13:41:32.654677 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:32.654712 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260924 13:41:33.470381 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:33.470170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.65:58933)
0924 13:41:33.470300 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:33.470336 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260924 13:41:33.531240 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:33.531053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.66:40035)
0924 13:41:33.531167 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:33.531198 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260924 13:41:33.653481 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 10548
I20260924 13:41:33.655462 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:33.655268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39185 (local address 127.25.240.67:56467)
0924 13:41:33.655380 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:33.655425 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.126:39185: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":69,"threads_started":1}
I20260924 13:41:33.658179 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 10717
I20260924 13:41:33.662438 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 10849
2026-09-24T13:41:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58210 ms)
[----------] 30 tests from SecurityITest (230814 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:33 dist-test-slave-m5z1 krb5kdc[11037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:33 dist-test-slave-m5z1 krb5kdc[11037](info): set up 2 sockets
Sep 24 13:41:33 dist-test-slave-m5z1 krb5kdc[11037](info): commencing operation
krb5kdc: starting...
W20260924 13:41:34.705538 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:34 dist-test-slave-m5z1 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:41:34.825371 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:44405
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:44405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:34.904815 11050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:34.905077 11050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:34.905114 11050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:34.906263 11050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:34.906330 11050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:34.906355 11050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:34.906375 11050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:34.906399 11050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:34.907830 11050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:44405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:44405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:34.908114 11050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:34.908339 11050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:34.913175 11055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:34.913236 11056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:34.913415 11058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:34.913405 11050 server_base.cc:1034] running on GCE node
I20260924 13:41:34.913688 11050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:34.914036 11050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:34.915205 11050 hybrid_clock.cc:648] HybridClock initialized: now 1790257294915180 us; error 36 us; skew 500 ppm
Sep 24 13:41:34 dist-test-slave-m5z1 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257294, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:34.917716 11050 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:41:34.918172 11050 webserver.cc:460] Webserver started at http://127.25.240.126:37237/ using document root <none> and password file <none>
I20260924 13:41:34.918416 11050 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:34.918485 11050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:34.918628 11050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:34.919759 11050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "4aa43682f8284040847e2b3cb755ebe9"
format_stamp: "Formatted at 2026-09-24 13:41:34 on dist-test-slave-m5z1"
server_key: "3f969a56033fb69b6b8062b36e640d39"
server_key_iv: "b5414bcddfc6254d49340e86a5218727"
server_key_version: "encryptionkey@0"
I20260924 13:41:34.920135 11050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "4aa43682f8284040847e2b3cb755ebe9"
format_stamp: "Formatted at 2026-09-24 13:41:34 on dist-test-slave-m5z1"
server_key: "3f969a56033fb69b6b8062b36e640d39"
server_key_iv: "b5414bcddfc6254d49340e86a5218727"
server_key_version: "encryptionkey@0"
I20260924 13:41:34.921445 11050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 13:41:34.922250 11065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:34.922422 11050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:41:34.922521 11050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "4aa43682f8284040847e2b3cb755ebe9"
format_stamp: "Formatted at 2026-09-24 13:41:34 on dist-test-slave-m5z1"
server_key: "3f969a56033fb69b6b8062b36e640d39"
server_key_iv: "b5414bcddfc6254d49340e86a5218727"
server_key_version: "encryptionkey@0"
I20260924 13:41:34.922603 11050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:34.936076 11050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:34.938459 11050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:34.938624 11050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:34.942662 11050 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:44405
I20260924 13:41:34.942710 11117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:44405 every 8 connection(s)
I20260924 13:41:34.943073 11050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:34.943930 11118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:34.946254 11118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9: Bootstrap starting.
I20260924 13:41:34.946841 11118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:34.947121 11118 log.cc:824] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:34.947974 11118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9: No bootstrap required, opened a new log
I20260924 13:41:34.949687 11118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa43682f8284040847e2b3cb755ebe9" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44405 } }
I20260924 13:41:34.949824 11118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:34.949854 11118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aa43682f8284040847e2b3cb755ebe9, State: Initialized, Role: FOLLOWER
I20260924 13:41:34.950047 11118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa43682f8284040847e2b3cb755ebe9" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44405 } }
I20260924 13:41:34.950130 11118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:34.950152 11118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:34.950212 11118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:34.950192 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 11050
I20260924 13:41:34.950314 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:34.950542 26561 external_mini_cluster.cc:1428] Setting key 15bcb07c29159cb141aa4899444e2713
I20260924 13:41:34.950834 11118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa43682f8284040847e2b3cb755ebe9" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44405 } }
I20260924 13:41:34.950968 11118 leader_election.cc:302] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aa43682f8284040847e2b3cb755ebe9; no voters:
I20260924 13:41:34.951126 11118 leader_election.cc:288] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:34.951170 11123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:34.951325 11123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [term 1 LEADER]: Becoming Leader. State: Replica: 4aa43682f8284040847e2b3cb755ebe9, State: Running, Role: LEADER
I20260924 13:41:34.951366 11118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:34.951486 11123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa43682f8284040847e2b3cb755ebe9" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44405 } }
I20260924 13:41:34.952066 11125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aa43682f8284040847e2b3cb755ebe9. Latest consensus state: current_term: 1 leader_uuid: "4aa43682f8284040847e2b3cb755ebe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa43682f8284040847e2b3cb755ebe9" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44405 } } }
I20260924 13:41:34.952163 11125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:34.952390 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aa43682f8284040847e2b3cb755ebe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aa43682f8284040847e2b3cb755ebe9" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44405 } } }
I20260924 13:41:34.952527 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:34.952723 11132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:34.953294 11132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:34.954946 11132 catalog_manager.cc:1347] Generated new cluster ID: 550053a0cedf466ba3a028000bc3bc85
I20260924 13:41:34.954998 11132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:41:34 dist-test-slave-m5z1 krb5kdc[11037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:34.963016 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:34.951006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.25.240.126:44405)
0924 13:41:34.951341 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:34.951351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:34.951402 (+ 51us) server_negotiation.cc:407] Connection header received
0924 13:41:34.951520 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:34.951527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:34.951611 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:34.951751 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:34.952729 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:34.954160 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:34.955211 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:34.955819 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:34.955949 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:34.958725 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:34.958729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:34.958729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:34.958740 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:34.961186 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:34.961644 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:34.961648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:34.961648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:34.961769 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:34.961991 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:34.961992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:34.961992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:34.962279 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:34.962435 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:34.962558 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 13:41:34.962794 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":94,"threads_started":1}
I20260924 13:41:34.966545 11132 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:34.967248 11132 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:34.976073 11132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9: Generated new TSK 0
I20260924 13:41:34.976264 11132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:41:35.004521 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44405
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:35.084993 11144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:35.085189 11144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:35.085211 11144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:35.086400 11144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:35.086439 11144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:35.086467 11144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:41:35.087885 11144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:35.088126 11144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:35.088351 11144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:35.093282 11149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.093274 11150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.093274 11152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.093459 11144 server_base.cc:1034] running on GCE node
I20260924 13:41:35.093750 11144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:35.094009 11144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:35.095156 11144 hybrid_clock.cc:648] HybridClock initialized: now 1790257295095147 us; error 30 us; skew 500 ppm
Sep 24 13:41:35 dist-test-slave-m5z1 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257295, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:35.097754 11144 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:41:35.098206 11144 webserver.cc:460] Webserver started at http://127.25.240.65:44283/ using document root <none> and password file <none>
I20260924 13:41:35.098433 11144 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:35.098501 11144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:35.098635 11144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:35.099646 11144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "6bf06855cf2546b18ca354b9ba622504"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "9d80c2d53b4ccbcc2682c63c1ce5f0bf"
server_key_iv: "d7afbb60880a6ce29d4a3374ca7028ea"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.100029 11144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "6bf06855cf2546b18ca354b9ba622504"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "9d80c2d53b4ccbcc2682c63c1ce5f0bf"
server_key_iv: "d7afbb60880a6ce29d4a3374ca7028ea"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.101328 11144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:35.102165 11159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.102355 11144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:41:35.102452 11144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "6bf06855cf2546b18ca354b9ba622504"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "9d80c2d53b4ccbcc2682c63c1ce5f0bf"
server_key_iv: "d7afbb60880a6ce29d4a3374ca7028ea"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.102532 11144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:35.115912 11144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:35.117871 11144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:35.118021 11144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:35.118244 11144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:35.118541 11144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:35.118594 11144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.118635 11144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:35.118664 11144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.124804 11144 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:44075
I20260924 13:41:35.124847 11272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:44075 every 8 connection(s)
I20260924 13:41:35.125204 11144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:41:35.129165 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 11144
I20260924 13:41:35.129285 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:41:35.129491 26561 external_mini_cluster.cc:1428] Setting key b7aae8ff1166e1e60ca8ec1636cfda95
Sep 24 13:41:35 dist-test-slave-m5z1 krb5kdc[11037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257295, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:35.135088 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.125933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:52687 (local address 127.25.240.126:44405)
0924 13:41:35.125997 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.125999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.126165 (+ 166us) server_negotiation.cc:407] Connection header received
0924 13:41:35.126678 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.126679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.126689 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.126751 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:35.127650 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.128169 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.129356 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.129707 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.129791 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.132215 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:35.132216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:35.132217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:35.132224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:35.133699 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:35.134090 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:35.134091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:35.134092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:35.134149 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:35.134410 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:35.134411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:35.134411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:35.134525 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:35.134592 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.134894 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.134997 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:41:35.135273 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.125901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:44405 (local address 127.25.240.65:52687)
0924 13:41:35.126049 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.126088 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.126589 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.126804 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.126807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.127354 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:35.127544 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.127548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.128295 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.128296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.129245 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.129248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.129841 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.129842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.129982 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.130553 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:35.130561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:35.132117 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:35.133783 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:35.133789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:35.133791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:35.134015 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:35.134223 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:35.134224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:35.134224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:35.134330 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:35.134612 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:35.134619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:35.134755 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.134872 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.135035 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260924 13:41:35.135816 11273 heartbeater.cc:347] Connected to a master server at 127.25.240.126:44405
I20260924 13:41:35.135896 11273 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:35.136089 11273 heartbeater.cc:511] Master 127.25.240.126:44405 requested a full tablet report, sending...
I20260924 13:41:35.136663 11082 ts_manager.cc:194] Registered new tserver with Master: 6bf06855cf2546b18ca354b9ba622504 (127.25.240.65:44075)
I20260924 13:41:35.137672 11082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:52687
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:41:35.157009 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44405
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:35.237221 11278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:35.237418 11278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:35.237442 11278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:35.238507 11278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:35.238541 11278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:35.238567 11278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:41:35.239892 11278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:35.240070 11278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:35.240228 11278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:35.245217 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.245188 11278 server_base.cc:1034] running on GCE node
W20260924 13:41:35.245219 11284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.245180 11286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.245524 11278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:35.245782 11278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:35.246961 11278 hybrid_clock.cc:648] HybridClock initialized: now 1790257295246942 us; error 33 us; skew 500 ppm
Sep 24 13:41:35 dist-test-slave-m5z1 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257295, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:35.249332 11278 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:41:35.249766 11278 webserver.cc:460] Webserver started at http://127.25.240.66:46363/ using document root <none> and password file <none>
I20260924 13:41:35.249955 11278 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:35.249996 11278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:35.250080 11278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:35.251015 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "1d793b800cae4006ab5e91ce656c0a10"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "ae60590b75977170708b586f2a89534a"
server_key_iv: "af86d8446fbb72465c979e3c227ad18e"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.251408 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "1d793b800cae4006ab5e91ce656c0a10"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "ae60590b75977170708b586f2a89534a"
server_key_iv: "af86d8446fbb72465c979e3c227ad18e"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.252767 11278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:35.253508 11293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.253722 11278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:35.253814 11278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "1d793b800cae4006ab5e91ce656c0a10"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "ae60590b75977170708b586f2a89534a"
server_key_iv: "af86d8446fbb72465c979e3c227ad18e"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.253886 11278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:35.265023 11278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:35.266904 11278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:35.267056 11278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:35.267325 11278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:35.267633 11278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:35.267666 11278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.267688 11278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:35.267735 11278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.273687 11278 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:35485
I20260924 13:41:35.273762 11406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:35485 every 8 connection(s)
I20260924 13:41:35.274088 11278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:41:35 dist-test-slave-m5z1 krb5kdc[11037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257295, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:35.281776 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 11278
I20260924 13:41:35.281917 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:41:35.282133 26561 external_mini_cluster.cc:1428] Setting key 844a73215fbd5b5a5aa1724500a37960
I20260924 13:41:35.283529 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.274881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:55673 (local address 127.25.240.126:44405)
0924 13:41:35.274932 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.274934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.275133 (+ 199us) server_negotiation.cc:407] Connection header received
0924 13:41:35.275681 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.275683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.275696 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.275746 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:35.276721 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.277239 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.278277 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.278496 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.278585 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.280643 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:35.280645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:35.280645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:35.280653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:35.282035 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:35.282534 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:35.282536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:35.282536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:35.282605 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:35.282879 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:35.282880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:35.282880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:35.283002 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:35.283079 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.283349 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.283438 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:41:35.283756 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.274826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:44405 (local address 127.25.240.66:55673)
0924 13:41:35.274997 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.275042 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.275553 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.275814 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.275817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.276409 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:35.276597 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.276602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.277342 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.277343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.278175 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.278177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.278593 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.278594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.278722 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.279174 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:35.279182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:35.280542 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:35.282122 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:35.282129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:35.282132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:35.282431 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:35.282676 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:35.282677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:35.282677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:35.282820 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:35.283070 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:35.283076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:35.283218 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.283335 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.283492 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260924 13:41:35.284240 11407 heartbeater.cc:347] Connected to a master server at 127.25.240.126:44405
I20260924 13:41:35.284381 11407 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:35.284576 11407 heartbeater.cc:511] Master 127.25.240.126:44405 requested a full tablet report, sending...
I20260924 13:41:35.284976 11082 ts_manager.cc:194] Registered new tserver with Master: 1d793b800cae4006ab5e91ce656c0a10 (127.25.240.66:35485)
I20260924 13:41:35.285342 11082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:55673
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:41:35.308641 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44405
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:43117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:35.389683 11412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:35.389884 11412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:35.389907 11412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:35.390987 11412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:35.391024 11412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:35.391050 11412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:41:35.392398 11412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:43117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:44405
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:35.392576 11412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:35.392750 11412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:41:35.398123 11412 server_base.cc:1034] running on GCE node
W20260924 13:41:35.398082 11420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.398082 11417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.398059 11418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.398488 11412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:35.398727 11412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:35.399887 11412 hybrid_clock.cc:648] HybridClock initialized: now 1790257295399861 us; error 39 us; skew 500 ppm
Sep 24 13:41:35 dist-test-slave-m5z1 krb5kdc[11037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257295, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:35.402415 11412 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:41:35.402849 11412 webserver.cc:460] Webserver started at http://127.25.240.67:35725/ using document root <none> and password file <none>
I20260924 13:41:35.403137 11412 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:35.403208 11412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:35.403362 11412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:35.404436 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "725cad8e66bf474ba9a4c19df35bd0c8"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "1078a31ba03908b17050dcd1ec4d50b6"
server_key_iv: "3fc703a74ed04420e3844a258379bcc5"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.404829 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "725cad8e66bf474ba9a4c19df35bd0c8"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "1078a31ba03908b17050dcd1ec4d50b6"
server_key_iv: "3fc703a74ed04420e3844a258379bcc5"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.406282 11412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:35.407088 11427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.407282 11412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:41:35.407371 11412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "725cad8e66bf474ba9a4c19df35bd0c8"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "1078a31ba03908b17050dcd1ec4d50b6"
server_key_iv: "3fc703a74ed04420e3844a258379bcc5"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.407457 11412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:35.422585 11412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:35.424999 11412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:35.425158 11412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:35.425400 11412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:35.425928 11412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:35.426002 11412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.426039 11412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:35.426069 11412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.431957 11412 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:34183
I20260924 13:41:35.432000 11540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:34183 every 8 connection(s)
I20260924 13:41:35.432312 11412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:41:35.433526 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 11412
I20260924 13:41:35.433668 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:41:35.433856 26561 external_mini_cluster.cc:1428] Setting key 3a5289318a13229b5a7af6fbc6677a9c
Sep 24 13:41:35 dist-test-slave-m5z1 krb5kdc[11037](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257295, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:35.441933 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.433035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:34569 (local address 127.25.240.126:44405)
0924 13:41:35.433099 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.433102 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.433340 (+ 238us) server_negotiation.cc:407] Connection header received
0924 13:41:35.434012 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.434014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.434026 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.434081 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:35.435252 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.435761 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.436989 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.437189 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.437256 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.439411 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:35.439413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:35.439413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:35.439421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:35.440661 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:35.441037 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:35.441038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:35.441038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:35.441089 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:35.441304 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:35.441304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:35.441304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:35.441409 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:35.441460 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.441770 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.441862 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:41:35.442078 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.433026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:44405 (local address 127.25.240.67:34569)
0924 13:41:35.433217 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.433253 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.433868 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.434175 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.434180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.434942 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:35.435142 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.435147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.435844 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.435845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.436856 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.436859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.437272 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.437273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.437378 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.437938 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:35.437946 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:35.439322 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:35.440736 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:35.440742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:35.440744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:35.440967 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:35.441150 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:35.441151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:35.441151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:35.441248 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:35.441465 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:35.441469 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:35.441631 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.441722 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.441843 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260924 13:41:35.442589 11541 heartbeater.cc:347] Connected to a master server at 127.25.240.126:44405
I20260924 13:41:35.442706 11541 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:35.442916 11541 heartbeater.cc:511] Master 127.25.240.126:44405 requested a full tablet report, sending...
I20260924 13:41:35.443419 11082 ts_manager.cc:194] Registered new tserver with Master: 725cad8e66bf474ba9a4c19df35bd0c8 (127.25.240.67:34183)
I20260924 13:41:35.443794 11082 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:34569
I20260924 13:41:35.445981 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:41:35.453332 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.447328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.25.240.126:44405)
0924 13:41:35.447392 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.447394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.447454 (+ 60us) server_negotiation.cc:407] Connection header received
0924 13:41:35.447506 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.447507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.447514 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.447565 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:35.448108 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.448580 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.449338 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.449533 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.449672 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.450793 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:35.450794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:35.450794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:35.450800 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:35.452200 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:35.452528 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:35.452528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:35.452529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:35.452608 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:35.452832 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:35.452832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:35.452833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:35.452982 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:35.453060 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.453214 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.453273 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:41:35.455024 11082 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:41:35.457192 11082 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:41:35.464051 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.459641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60468 (local address 127.25.240.67:34183)
0924 13:41:35.459797 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.459809 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.459821 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:41:35.459870 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.459872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.459888 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.460169 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:35.460245 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.461326 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.462976 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.463680 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.463756 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.463809 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.463923 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.463966 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260924 13:41:35.464049 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.459379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34183 (local address 127.0.0.1:60468)
0924 13:41:35.459553 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.459618 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.459685 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.460012 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.460014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.460017 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:35.460085 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.460087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.461431 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.461432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.462904 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.462913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.463756 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.463757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.463813 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.463822 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.463904 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.463966 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":53,"thread_start_us":67,"threads_started":1}
I20260924 13:41:35.464893 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.459379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:44075 (local address 127.0.0.1:33348)
0924 13:41:35.459787 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.459802 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.459843 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.460012 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.460018 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.460021 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:35.460100 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.460102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.461432 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.461433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.462904 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.462907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.464672 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.464673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.464739 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.464743 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.464791 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.464831 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":92,"thread_start_us":66,"threads_started":1}
I20260924 13:41:35.465047 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.459550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.25.240.65:44075)
0924 13:41:35.459845 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.459848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.459886 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:41:35.459925 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.459927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.459941 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.460016 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:35.460168 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.461309 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.463850 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.464587 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.464650 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.464701 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.464823 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.464867 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":93,"threads_started":1}
I20260924 13:41:35.465271 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.459407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:35485 (local address 127.0.0.1:54022)
0924 13:41:35.459833 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.459843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.459885 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.461680 (+ 1795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.461681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.461683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:35.461737 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.461739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.462714 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.462715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.463856 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.463859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.465041 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.465042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.465111 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.465115 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.465162 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.465205 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260924 13:41:35.465193 11475 tablet_service.cc:1467] Processing CreateTablet for tablet 08574799f3a642cd9db8edc004e0fd87 (DEFAULT_TABLE table=test-table [id=7d568c248a554a69aa162e437014ec93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:35.465595 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08574799f3a642cd9db8edc004e0fd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:35.465617 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.459667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.25.240.66:35485)
0924 13:41:35.461507 (+ 1840us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.461510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.461518 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:41:35.461547 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.461549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.461607 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.461681 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:35.461794 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.462614 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.463947 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.464894 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.464957 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.465003 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.465455 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.465505 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1808,"thread_start_us":86,"threads_started":1}
I20260924 13:41:35.466141 11341 tablet_service.cc:1467] Processing CreateTablet for tablet 08574799f3a642cd9db8edc004e0fd87 (DEFAULT_TABLE table=test-table [id=7d568c248a554a69aa162e437014ec93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:35.466229 11207 tablet_service.cc:1467] Processing CreateTablet for tablet 08574799f3a642cd9db8edc004e0fd87 (DEFAULT_TABLE table=test-table [id=7d568c248a554a69aa162e437014ec93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:35.466467 11341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08574799f3a642cd9db8edc004e0fd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:35.466567 11207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08574799f3a642cd9db8edc004e0fd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:35.468807 11555 tablet_bootstrap.cc:492] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Bootstrap starting.
I20260924 13:41:35.469116 11556 tablet_bootstrap.cc:492] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: Bootstrap starting.
I20260924 13:41:35.469481 11555 tablet_bootstrap.cc:654] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:35.469816 11556 tablet_bootstrap.cc:654] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:35.469970 11557 tablet_bootstrap.cc:492] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504: Bootstrap starting.
I20260924 13:41:35.470176 11555 log.cc:824] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:35.470402 11556 log.cc:824] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:35.470727 11557 tablet_bootstrap.cc:654] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:35.471127 11555 tablet_bootstrap.cc:492] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: No bootstrap required, opened a new log
I20260924 13:41:35.471133 11556 tablet_bootstrap.cc:492] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: No bootstrap required, opened a new log
I20260924 13:41:35.471199 11556 ts_tablet_manager.cc:1397] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:41:35.471200 11555 ts_tablet_manager.cc:1397] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:41:35.471218 11557 log.cc:824] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:35.471853 11557 tablet_bootstrap.cc:492] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504: No bootstrap required, opened a new log
I20260924 13:41:35.471930 11557 ts_tablet_manager.cc:1397] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:41:35.473152 11555 raft_consensus.cc:348] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.473151 11556 raft_consensus.cc:348] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.473210 11557 raft_consensus.cc:348] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.473338 11555 raft_consensus.cc:374] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:35.473357 11557 raft_consensus.cc:374] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:35.473338 11556 raft_consensus.cc:374] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:35.473378 11555 raft_consensus.cc:729] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d793b800cae4006ab5e91ce656c0a10, State: Initialized, Role: FOLLOWER
I20260924 13:41:35.473407 11556 raft_consensus.cc:729] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725cad8e66bf474ba9a4c19df35bd0c8, State: Initialized, Role: FOLLOWER
I20260924 13:41:35.473405 11557 raft_consensus.cc:729] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf06855cf2546b18ca354b9ba622504, State: Initialized, Role: FOLLOWER
I20260924 13:41:35.473505 11555 consensus_queue.cc:260] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.473505 11556 consensus_queue.cc:260] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.473541 11557 consensus_queue.cc:260] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.473805 11555 ts_tablet_manager.cc:1428] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:41:35.473805 11556 ts_tablet_manager.cc:1428] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:41:35.473852 11557 ts_tablet_manager.cc:1428] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:41:35.473948 11541 heartbeater.cc:503] Master 127.25.240.126:44405 was elected leader, sending a full tablet report...
I20260924 13:41:35.473989 11273 heartbeater.cc:503] Master 127.25.240.126:44405 was elected leader, sending a full tablet report...
I20260924 13:41:35.474210 11407 heartbeater.cc:503] Master 127.25.240.126:44405 was elected leader, sending a full tablet report...
I20260924 13:41:35.501029 11562 raft_consensus.cc:482] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:35.501168 11562 raft_consensus.cc:504] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.501591 11562 leader_election.cc:288] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 725cad8e66bf474ba9a4c19df35bd0c8 (127.25.240.67:34183), 6bf06855cf2546b18ca354b9ba622504 (127.25.240.65:44075)
I20260924 13:41:35.505512 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.501822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:43603 (local address 127.25.240.67:34183)
0924 13:41:35.501863 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.501866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.502001 (+ 135us) server_negotiation.cc:407] Connection header received
0924 13:41:35.502050 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.502051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.502059 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.502113 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:35.502303 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.503045 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.504558 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.505117 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.505199 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.505209 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.505405 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.505455 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:41:35.505512 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.501815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:39827 (local address 127.25.240.65:44075)
0924 13:41:35.501863 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.501866 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.501921 (+ 55us) server_negotiation.cc:407] Connection header received
0924 13:41:35.502022 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.502023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.502033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.502101 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:35.502303 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.503051 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.504558 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.505166 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.505229 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.505236 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.505405 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.505442 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:41:35.505527 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.501756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:44075 (local address 127.25.240.66:39827)
0924 13:41:35.501868 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.501881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.501938 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.502118 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.502119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.502120 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:35.502223 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.502225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.503134 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.503135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.504470 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.504473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.505234 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.505235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.505322 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.505328 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.505394 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.505439 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":85}
I20260924 13:41:35.505533 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.501756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34183 (local address 127.25.240.66:43603)
0924 13:41:35.501935 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.501944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.501994 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.502122 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.502123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.502125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:35.502231 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.502232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.503134 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.503135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.504470 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.504473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.505234 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.505235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.505323 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.505328 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.505388 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.505439 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":90,"thread_start_us":105,"threads_started":1}
I20260924 13:41:35.505885 11227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08574799f3a642cd9db8edc004e0fd87" candidate_uuid: "1d793b800cae4006ab5e91ce656c0a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf06855cf2546b18ca354b9ba622504" is_pre_election: true
I20260924 13:41:35.505975 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08574799f3a642cd9db8edc004e0fd87" candidate_uuid: "1d793b800cae4006ab5e91ce656c0a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" is_pre_election: true
I20260924 13:41:35.506054 11227 raft_consensus.cc:2454] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d793b800cae4006ab5e91ce656c0a10 in term 0.
I20260924 13:41:35.506131 11495 raft_consensus.cc:2454] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d793b800cae4006ab5e91ce656c0a10 in term 0.
I20260924 13:41:35.506263 11297 leader_election.cc:302] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d793b800cae4006ab5e91ce656c0a10, 6bf06855cf2546b18ca354b9ba622504; no voters:
I20260924 13:41:35.506381 11562 raft_consensus.cc:2781] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:41:35.506466 11562 raft_consensus.cc:482] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:41:35.506496 11562 raft_consensus.cc:3037] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:35.507206 11562 raft_consensus.cc:504] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.507364 11562 leader_election.cc:288] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [CANDIDATE]: Term 1 election: Requested vote from peers 725cad8e66bf474ba9a4c19df35bd0c8 (127.25.240.67:34183), 6bf06855cf2546b18ca354b9ba622504 (127.25.240.65:44075)
I20260924 13:41:35.507500 11495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08574799f3a642cd9db8edc004e0fd87" candidate_uuid: "1d793b800cae4006ab5e91ce656c0a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "725cad8e66bf474ba9a4c19df35bd0c8"
I20260924 13:41:35.507557 11227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08574799f3a642cd9db8edc004e0fd87" candidate_uuid: "1d793b800cae4006ab5e91ce656c0a10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf06855cf2546b18ca354b9ba622504"
I20260924 13:41:35.507622 11495 raft_consensus.cc:3037] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:35.507666 11227 raft_consensus.cc:3037] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:35.508415 11227 raft_consensus.cc:2454] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d793b800cae4006ab5e91ce656c0a10 in term 1.
I20260924 13:41:35.508420 11495 raft_consensus.cc:2454] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d793b800cae4006ab5e91ce656c0a10 in term 1.
I20260924 13:41:35.508598 11297 leader_election.cc:302] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d793b800cae4006ab5e91ce656c0a10, 6bf06855cf2546b18ca354b9ba622504; no voters:
I20260924 13:41:35.508678 11562 raft_consensus.cc:2781] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:35.508826 11562 raft_consensus.cc:686] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 1 LEADER]: Becoming Leader. State: Replica: 1d793b800cae4006ab5e91ce656c0a10, State: Running, Role: LEADER
I20260924 13:41:35.508924 11562 consensus_queue.cc:237] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } }
I20260924 13:41:35.509773 11080 catalog_manager.cc:5495] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d793b800cae4006ab5e91ce656c0a10 (127.25.240.66). New cstate: current_term: 1 leader_uuid: "1d793b800cae4006ab5e91ce656c0a10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d793b800cae4006ab5e91ce656c0a10" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 35485 } health_report { overall_health: HEALTHY } } }
I20260924 13:41:35.524029 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.521855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.25.240.66:35485)
0924 13:41:35.521901 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.521902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.521946 (+ 44us) server_negotiation.cc:407] Connection header received
0924 13:41:35.522012 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.522013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.522030 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.522088 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:41:35.522205 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.522667 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.523375 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.523569 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.523624 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.523748 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:41:35.523843 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:41:35.523902 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.523940 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.523957 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 13:41:35.524610 11408 tablet.cc:2369] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:41:35.525861 11495 raft_consensus.cc:1261] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 1 FOLLOWER]: Refusing update from remote peer 1d793b800cae4006ab5e91ce656c0a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:35.526180 11227 raft_consensus.cc:1261] T 08574799f3a642cd9db8edc004e0fd87 P 6bf06855cf2546b18ca354b9ba622504 [term 1 FOLLOWER]: Refusing update from remote peer 1d793b800cae4006ab5e91ce656c0a10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:35.526196 11565 consensus_queue.cc:1035] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "725cad8e66bf474ba9a4c19df35bd0c8" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:35.526511 11565 consensus_queue.cc:1035] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf06855cf2546b18ca354b9ba622504" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:35.528769 11569 mvcc.cc:205] Tried to move back new op lower bound from 7332893882471370752 to 7332893882405212160. Current Snapshot: MvccSnapshot[applied={T|T < 7332893882471370752}]
I20260924 13:41:35.528903 11570 mvcc.cc:205] Tried to move back new op lower bound from 7332893882471370752 to 7332893882405212160. Current Snapshot: MvccSnapshot[applied={T|T < 7332893882471370752}]
I20260924 13:41:35.530941 11080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:35.531029 11080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:35.531760 11080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4aa43682f8284040847e2b3cb755ebe9: Sending DeleteTablet for 3 replicas of tablet 08574799f3a642cd9db8edc004e0fd87
I20260924 13:41:35.532043 11475 tablet_service.cc:1514] Processing DeleteTablet for tablet 08574799f3a642cd9db8edc004e0fd87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:35 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:60468
I20260924 13:41:35.532127 11207 tablet_service.cc:1514] Processing DeleteTablet for tablet 08574799f3a642cd9db8edc004e0fd87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:35 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:33348
I20260924 13:41:35.532171 11341 tablet_service.cc:1514] Processing DeleteTablet for tablet 08574799f3a642cd9db8edc004e0fd87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:35 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:54022
I20260924 13:41:35.532343 11580 tablet_replica.cc:330] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: stopping tablet replica
I20260924 13:41:35.532383 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 11144
I20260924 13:41:35.532435 11580 raft_consensus.cc:2229] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:41:35.533084 11578 tablet_replica.cc:330] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: stopping tablet replica
I20260924 13:41:35.533151 11578 raft_consensus.cc:2229] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:35.533214 11578 raft_consensus.cc:2258] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:35.533381 11580 raft_consensus.cc:2258] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:35.533692 11578 ts_tablet_manager.cc:1905] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:35.533885 11580 ts_tablet_manager.cc:1905] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:35.535462 11580 ts_tablet_manager.cc:1918] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:35.535561 11580 log.cc:1196] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/08574799f3a642cd9db8edc004e0fd87
I20260924 13:41:35.535570 11578 ts_tablet_manager.cc:1918] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:35.535627 11578 log.cc:1196] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/08574799f3a642cd9db8edc004e0fd87
I20260924 13:41:35.535853 11580 ts_tablet_manager.cc:1939] T 08574799f3a642cd9db8edc004e0fd87 P 1d793b800cae4006ab5e91ce656c0a10: Deleting consensus metadata
I20260924 13:41:35.535893 11578 ts_tablet_manager.cc:1939] T 08574799f3a642cd9db8edc004e0fd87 P 725cad8e66bf474ba9a4c19df35bd0c8: Deleting consensus metadata
I20260924 13:41:35.536324 11068 catalog_manager.cc:4841] TS 725cad8e66bf474ba9a4c19df35bd0c8 (127.25.240.67:34183): tablet 08574799f3a642cd9db8edc004e0fd87 (table test-table [id=7d568c248a554a69aa162e437014ec93]) successfully deleted
I20260924 13:41:35.536386 11066 catalog_manager.cc:4841] TS 1d793b800cae4006ab5e91ce656c0a10 (127.25.240.66:35485): tablet 08574799f3a642cd9db8edc004e0fd87 (table test-table [id=7d568c248a554a69aa162e437014ec93]) successfully deleted
W20260924 13:41:35.537655 11069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:44075 (error 108)
I20260924 13:41:35.537926 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 11278
I20260924 13:41:35.538096 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.537983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:44075 (local address 127.0.0.1:33356)
0924 13:41:35.538031 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.538066 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:44075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:41:35.538543 11069 catalog_manager.cc:4584] TS 6bf06855cf2546b18ca354b9ba622504 (127.25.240.65:44075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 08574799f3a642cd9db8edc004e0fd87: Network error: Client connection negotiation failed: client connection to 127.25.240.65:44075: connect: Connection refused (error 111)
I20260924 13:41:35.545322 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 11412
I20260924 13:41:35.549551 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 11050
2026-09-24T13:41:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1887 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T13:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:35Z Disabled control of system clock
I20260924 13:41:35.576653 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35575
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:35.655619 11585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:35.655825 11585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:35.655845 11585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:35.656946 11585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:35.656983 11585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:35.656996 11585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:35.657012 11585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:35.657027 11585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:35.658288 11585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:35575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:35.658468 11585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:35.658685 11585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:35.663311 11590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.663358 11585 server_base.cc:1034] running on GCE node
W20260924 13:41:35.663288 11591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.663323 11593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.663805 11585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:35.664076 11585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:35.665222 11585 hybrid_clock.cc:648] HybridClock initialized: now 1790257295665209 us; error 27 us; skew 500 ppm
I20260924 13:41:35.666396 11585 webserver.cc:460] Webserver started at http://127.25.240.126:34519/ using document root <none> and password file <none>
I20260924 13:41:35.666616 11585 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:35.666661 11585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:35.666819 11585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:35.667833 11585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "123171e410a8488aaed21e2fbf47ce10"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "5edfe7fbe329e1adbb80252ebcc393bb"
server_key_iv: "02ec1112a456ac5aca84abbf9e666279"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.668196 11585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "123171e410a8488aaed21e2fbf47ce10"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "5edfe7fbe329e1adbb80252ebcc393bb"
server_key_iv: "02ec1112a456ac5aca84abbf9e666279"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.669445 11585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:35.670219 11599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.670368 11585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.670457 11585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "123171e410a8488aaed21e2fbf47ce10"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "5edfe7fbe329e1adbb80252ebcc393bb"
server_key_iv: "02ec1112a456ac5aca84abbf9e666279"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.670536 11585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:35.683827 11585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:35.684099 11585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:35.684235 11585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:35.687939 11585 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:35575
I20260924 13:41:35.688019 11651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:35575 every 8 connection(s)
I20260924 13:41:35.688313 11585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:35.689253 11652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:35.690572 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 11585
I20260924 13:41:35.690678 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:35.690861 26561 external_mini_cluster.cc:1428] Setting key 74f5cdd1c903cb8791aa0f0496e9b991
I20260924 13:41:35.691973 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10: Bootstrap starting.
I20260924 13:41:35.692685 11652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:35.693005 11652 log.cc:824] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:35.693719 11652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10: No bootstrap required, opened a new log
I20260924 13:41:35.696167 11652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123171e410a8488aaed21e2fbf47ce10" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35575 } }
I20260924 13:41:35.696336 11652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:35.696372 11652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123171e410a8488aaed21e2fbf47ce10, State: Initialized, Role: FOLLOWER
I20260924 13:41:35.696506 11652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123171e410a8488aaed21e2fbf47ce10" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35575 } }
I20260924 13:41:35.696576 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.691346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43676 (local address 127.25.240.126:35575)
0924 13:41:35.691555 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.691564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.691597 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:41:35.692100 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.692108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.692182 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.692346 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:35.692490 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.693820 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.694578 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.695217 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.695377 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.695887 (+ 510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:35.695892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:35.695893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:35.696114 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:35.696157 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:35.696171 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:35.696249 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.696289 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.696374 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260924 13:41:35.696588 11652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:35.696679 11652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:35.696725 11652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:35.697336 11652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123171e410a8488aaed21e2fbf47ce10" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35575 } }
I20260924 13:41:35.697423 11652 leader_election.cc:302] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 123171e410a8488aaed21e2fbf47ce10; no voters:
I20260924 13:41:35.697541 11652 leader_election.cc:288] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:35.697737 11657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:35.697861 11652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:35.697870 11657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [term 1 LEADER]: Becoming Leader. State: Replica: 123171e410a8488aaed21e2fbf47ce10, State: Running, Role: LEADER
I20260924 13:41:35.697973 11657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123171e410a8488aaed21e2fbf47ce10" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35575 } }
I20260924 13:41:35.698347 11659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 123171e410a8488aaed21e2fbf47ce10. Latest consensus state: current_term: 1 leader_uuid: "123171e410a8488aaed21e2fbf47ce10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123171e410a8488aaed21e2fbf47ce10" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35575 } } }
I20260924 13:41:35.698361 11658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "123171e410a8488aaed21e2fbf47ce10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "123171e410a8488aaed21e2fbf47ce10" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35575 } } }
I20260924 13:41:35.698596 11658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:35.698582 11659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:35.698976 11667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:35.699476 11667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:35.700871 11667 catalog_manager.cc:1347] Generated new cluster ID: 6de075796d4a4ed1815b4ba3af64bbcc
I20260924 13:41:35.700923 11667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:35.705685 11667 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:35.706161 11667 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:35.710161 11667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10: Generated new TSK 0
I20260924 13:41:35.710322 11667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:35.714958 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35575
--builtin_ntp_servers=127.25.240.84:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:35.795461 11677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:35.795687 11677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:35.795725 11677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:35.796855 11677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:35.796916 11677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:35.796959 11677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:41:35.798523 11677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:35.798772 11677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:35.799077 11677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:35.804039 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.804039 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.804251 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.804368 11677 server_base.cc:1034] running on GCE node
I20260924 13:41:35.804558 11677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:35.804798 11677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:35.805949 11677 hybrid_clock.cc:648] HybridClock initialized: now 1790257295805933 us; error 33 us; skew 500 ppm
I20260924 13:41:35.807134 11677 webserver.cc:460] Webserver started at http://127.25.240.65:37871/ using document root <none> and password file <none>
I20260924 13:41:35.807381 11677 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:35.807449 11677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:35.807605 11677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:35.808629 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "fef987a2ef1f4c8780b791c490086c1f"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "591d2304c45291dbca5dd896909a34eb"
server_key_iv: "1a3c56dea620f6aaa01b69d458077d7c"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.808970 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "fef987a2ef1f4c8780b791c490086c1f"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "591d2304c45291dbca5dd896909a34eb"
server_key_iv: "1a3c56dea620f6aaa01b69d458077d7c"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.810439 11677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:35.811185 11691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.811376 11677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:35.811480 11677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "fef987a2ef1f4c8780b791c490086c1f"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "591d2304c45291dbca5dd896909a34eb"
server_key_iv: "1a3c56dea620f6aaa01b69d458077d7c"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.811545 11677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:35.832042 11677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:35.832288 11677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:35.832386 11677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:35.832556 11677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:35.832897 11677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:35.832932 11677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.832984 11677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:35.833009 11677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.838383 11677 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:32787
I20260924 13:41:35.838457 11804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:32787 every 8 connection(s)
I20260924 13:41:35.838780 11677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:41:35.839967 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 11677
I20260924 13:41:35.840066 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:41:35.840257 26561 external_mini_cluster.cc:1428] Setting key 7337092eee78bbf1e077f2bcbab01ec1
I20260924 13:41:35.841127 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35575
--builtin_ntp_servers=127.25.240.84:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:41:35.844903 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.839562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:38409 (local address 127.25.240.126:35575)
0924 13:41:35.839641 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.839644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.839901 (+ 257us) server_negotiation.cc:407] Connection header received
0924 13:41:35.840839 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.840840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.840853 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.840908 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:35.841734 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.842406 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.843357 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.843688 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.843772 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.844585 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:35.844587 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:35.844587 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:35.844592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:35.844612 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:35.844616 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:35.844679 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.844735 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.844761 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 13:41:35.845058 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.839518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35575 (local address 127.25.240.65:38409)
0924 13:41:35.839744 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.839798 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.840502 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.840979 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.840983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.840993 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:35.841643 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.841649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.842506 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.842507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.843276 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.843279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.843782 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.843783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.843884 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.844456 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:35.844456 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:35.844482 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:35.844504 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:35.844679 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:35.844683 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:35.844685 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.844732 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.844805 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":93,"threads_started":1}
I20260924 13:41:35.845498 11805 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35575
I20260924 13:41:35.845618 11805 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:35.845793 11805 heartbeater.cc:511] Master 127.25.240.126:35575 requested a full tablet report, sending...
I20260924 13:41:35.846266 11616 ts_manager.cc:194] Registered new tserver with Master: fef987a2ef1f4c8780b791c490086c1f (127.25.240.65:32787)
I20260924 13:41:35.847014 11616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.65:38409
W20260924 13:41:35.924065 11808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:35.924279 11808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:35.924325 11808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:35.925521 11808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:35.925588 11808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:35.925631 11808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:41:35.926962 11808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:35.927204 11808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:35.927485 11808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:35.932384 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:35.932416 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.932440 11808 server_base.cc:1034] running on GCE node
W20260924 13:41:35.932396 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:35.932798 11808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:35.933043 11808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:35.934221 11808 hybrid_clock.cc:648] HybridClock initialized: now 1790257295934194 us; error 35 us; skew 500 ppm
I20260924 13:41:35.935514 11808 webserver.cc:460] Webserver started at http://127.25.240.66:38785/ using document root <none> and password file <none>
I20260924 13:41:35.935750 11808 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:35.935813 11808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:35.935940 11808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:35.937362 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "f511b6a3b77a48ce86990e5eacb3c61e"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "cf116480d30afbf230f535ab9dc1aad2"
server_key_iv: "92527b5dd71c4bc7f6e14ec574e4e547"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.937793 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "f511b6a3b77a48ce86990e5eacb3c61e"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "cf116480d30afbf230f535ab9dc1aad2"
server_key_iv: "92527b5dd71c4bc7f6e14ec574e4e547"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.939043 11808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:41:35.939755 11822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.939929 11808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:41:35.940021 11808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "f511b6a3b77a48ce86990e5eacb3c61e"
format_stamp: "Formatted at 2026-09-24 13:41:35 on dist-test-slave-m5z1"
server_key: "cf116480d30afbf230f535ab9dc1aad2"
server_key_iv: "92527b5dd71c4bc7f6e14ec574e4e547"
server_key_version: "encryptionkey@0"
I20260924 13:41:35.940099 11808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:35.964139 11808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:35.964498 11808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:35.964651 11808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:35.964893 11808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:35.965274 11808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:35.965344 11808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.965386 11808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:35.965413 11808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:35.971342 11808 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:39293
I20260924 13:41:35.971391 11935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:39293 every 8 connection(s)
I20260924 13:41:35.971756 11808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:41:35.976014 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 11808
I20260924 13:41:35.976114 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:41:35.976327 26561 external_mini_cluster.cc:1428] Setting key e53b4eaaf920d1d81adf1f81b7eb80f8
I20260924 13:41:35.976994 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.972490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:44723 (local address 127.25.240.126:35575)
0924 13:41:35.972536 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:35.972538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:35.972722 (+ 184us) server_negotiation.cc:407] Connection header received
0924 13:41:35.973227 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:35.973228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:35.973243 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:35.973304 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:35.974024 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.974561 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.975546 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.975729 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.975784 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.976736 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:35.976737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:35.976737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:35.976742 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:35.976758 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:35.976763 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:35.976820 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:35.976874 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 13:41:35.976895 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 13:41:35.977246 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:35.972428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35575 (local address 127.25.240.66:44723)
0924 13:41:35.972607 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:35.972647 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:35.973120 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:35.973362 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:35.973365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:35.973373 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:35.973939 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.973944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.974653 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.974654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.975465 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:35.975467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:35.975830 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:35.975831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:35.975998 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:35.976625 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:35.976626 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:35.976655 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:35.976676 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:35.976821 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:35.976823 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:35.976826 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:35.976874 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:41:35.976986 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 13:41:35.977526 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35575
--builtin_ntp_servers=127.25.240.84:44617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:41:35.977982 11936 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35575
I20260924 13:41:35.978077 11936 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:35.978250 11936 heartbeater.cc:511] Master 127.25.240.126:35575 requested a full tablet report, sending...
I20260924 13:41:35.978596 11616 ts_manager.cc:194] Registered new tserver with Master: f511b6a3b77a48ce86990e5eacb3c61e (127.25.240.66:39293)
I20260924 13:41:35.979072 11616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.66:44723
W20260924 13:41:36.057065 11939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:36.057264 11939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:36.057286 11939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:36.058380 11939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:36.058422 11939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:36.058446 11939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:41:36.059698 11939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:36.059899 11939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:36.060201 11939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:36.065129 11945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:36.065145 11947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:36.065249 11939 server_base.cc:1034] running on GCE node
W20260924 13:41:36.065251 11944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:36.065680 11939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:36.065912 11939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:36.067050 11939 hybrid_clock.cc:648] HybridClock initialized: now 1790257296067043 us; error 33 us; skew 500 ppm
I20260924 13:41:36.068243 11939 webserver.cc:460] Webserver started at http://127.25.240.67:35329/ using document root <none> and password file <none>
I20260924 13:41:36.068456 11939 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:36.068498 11939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:36.068586 11939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:36.069661 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "6328c47ebb5c4a1e98f950658faaf1aa"
format_stamp: "Formatted at 2026-09-24 13:41:36 on dist-test-slave-m5z1"
server_key: "f7ffd8e3ef39ce0f6697cc575786187b"
server_key_iv: "56e0190f964230c57ed7dfc3e363b4cd"
server_key_version: "encryptionkey@0"
I20260924 13:41:36.069969 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "6328c47ebb5c4a1e98f950658faaf1aa"
format_stamp: "Formatted at 2026-09-24 13:41:36 on dist-test-slave-m5z1"
server_key: "f7ffd8e3ef39ce0f6697cc575786187b"
server_key_iv: "56e0190f964230c57ed7dfc3e363b4cd"
server_key_version: "encryptionkey@0"
I20260924 13:41:36.071208 11939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:36.071921 11953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:36.072095 11939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:36.072182 11939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "6328c47ebb5c4a1e98f950658faaf1aa"
format_stamp: "Formatted at 2026-09-24 13:41:36 on dist-test-slave-m5z1"
server_key: "f7ffd8e3ef39ce0f6697cc575786187b"
server_key_iv: "56e0190f964230c57ed7dfc3e363b4cd"
server_key_version: "encryptionkey@0"
I20260924 13:41:36.072234 11939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:36.090749 11939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:36.091008 11939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:36.091107 11939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:36.091269 11939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:36.091596 11939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:36.091629 11939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:36.091666 11939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:36.091692 11939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:36.097450 11939 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:44201
I20260924 13:41:36.097494 12066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:44201 every 8 connection(s)
I20260924 13:41:36.097879 11939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:41:36.102583 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 11939
I20260924 13:41:36.102703 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:41:36.102934 26561 external_mini_cluster.cc:1428] Setting key ddd5f2c9c513e4254cbde67d7dac3251
I20260924 13:41:36.102958 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.098556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:33339 (local address 127.25.240.126:35575)
0924 13:41:36.098618 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:36.098620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:36.098839 (+ 219us) server_negotiation.cc:407] Connection header received
0924 13:41:36.099374 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:36.099375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:36.099385 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:36.099449 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:36.100070 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.100553 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.101510 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.101741 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.101792 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.102660 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:36.102661 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:36.102662 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:36.102668 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:36.102686 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:36.102691 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:36.102752 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:36.102842 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 13:41:36.102872 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:41:36.103188 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.098559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35575 (local address 127.25.240.67:33339)
0924 13:41:36.098722 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:36.098754 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:36.099286 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:36.099500 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:36.099503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:36.099512 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:36.100003 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.100007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.100635 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.100636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.101433 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.101435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.101806 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.101806 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.101941 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.102523 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:36.102524 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:36.102551 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:36.102571 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:36.102755 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:36.102760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:36.102764 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:36.102830 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:41:36.102913 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260924 13:41:36.103785 12067 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35575
I20260924 13:41:36.103907 12067 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:36.104089 12067 heartbeater.cc:511] Master 127.25.240.126:35575 requested a full tablet report, sending...
I20260924 13:41:36.104430 11616 ts_manager.cc:194] Registered new tserver with Master: 6328c47ebb5c4a1e98f950658faaf1aa (127.25.240.67:44201)
I20260924 13:41:36.104802 11616 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.67:33339
I20260924 13:41:36.114701 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:41:36.118850 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.116116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43682 (local address 127.25.240.126:35575)
0924 13:41:36.116162 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:36.116164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:36.116322 (+ 158us) server_negotiation.cc:407] Connection header received
0924 13:41:36.116345 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:36.116345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:36.116354 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:36.116401 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:36.116558 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.117031 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.117818 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.118003 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.118059 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.118608 (+ 549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:36.118609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:36.118609 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:36.118613 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:36.118625 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:36.118628 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:36.118669 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:36.118711 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 13:41:36.118729 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:41:36.120414 11616 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:41:36.122056 11616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:41:36.128167 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.124175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:44201 (local address 127.0.0.1:43524)
0924 13:41:36.124573 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:36.124580 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:36.124607 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:36.124825 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:36.124828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:36.124831 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:36.124900 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.124902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.125950 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.125951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.127074 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.127077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.127872 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.127873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.127950 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.127965 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 13:41:36.128039 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 13:41:36.128086 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":3,"thread_start_us":111,"threads_started":1}
I20260924 13:41:36.128178 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.124437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.25.240.67:44201)
0924 13:41:36.124590 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:36.124595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:36.124625 (+ 30us) server_negotiation.cc:407] Connection header received
0924 13:41:36.124670 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:36.124672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:36.124689 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:36.124983 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:36.125044 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.125860 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.127147 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.127785 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.127861 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.127951 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:36.128051 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 13:41:36.128089 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260924 13:41:36.128489 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.124174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32787 (local address 127.0.0.1:46820)
0924 13:41:36.124447 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:36.124457 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:36.124498 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:36.125038 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:36.125039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:36.125041 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:36.125070 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.125070 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.126406 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.126407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.127350 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.127352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.128256 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.128257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.128329 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.128333 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:36.128369 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 13:41:36.128402 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":108,"threads_started":1}
I20260924 13:41:36.128988 12001 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6de12c846a4056a733baeb92463dc5 (DEFAULT_TABLE table=test-table [id=9de8c515a7044509b51fcfabb0b55c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:36.129138 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.124617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.25.240.65:32787)
0924 13:41:36.124807 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:36.124811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:36.124821 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:41:36.124860 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:36.124862 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:36.124877 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:36.125139 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:36.125201 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.126316 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.127432 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.128148 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.128880 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.128991 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:36.129017 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 13:41:36.129055 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260924 13:41:36.129361 12001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6de12c846a4056a733baeb92463dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:36.129606 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.124164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:39293 (local address 127.0.0.1:41844)
0924 13:41:36.124345 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:36.124387 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:36.124467 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:36.124914 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:36.124915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:36.124917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:36.124972 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.124974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.127420 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.127420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.128200 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.128203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.129338 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.129339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.129402 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.129407 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:41:36.129455 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:41:36.129498 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":55,"threads_started":1}
I20260924 13:41:36.129951 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.124257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.25.240.66:39293)
0924 13:41:36.124729 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:36.124735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:36.124747 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:41:36.124789 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:36.124791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:36.124807 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:36.125054 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:36.125121 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.126315 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.128457 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.129244 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.129770 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.129836 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:36.129857 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 13:41:36.129887 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":94,"threads_started":1}
I20260924 13:41:36.130036 11739 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6de12c846a4056a733baeb92463dc5 (DEFAULT_TABLE table=test-table [id=9de8c515a7044509b51fcfabb0b55c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:36.130363 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6de12c846a4056a733baeb92463dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:36.130481 11870 tablet_service.cc:1467] Processing CreateTablet for tablet 5d6de12c846a4056a733baeb92463dc5 (DEFAULT_TABLE table=test-table [id=9de8c515a7044509b51fcfabb0b55c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:36.130788 11870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d6de12c846a4056a733baeb92463dc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:36.132894 12081 tablet_bootstrap.cc:492] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: Bootstrap starting.
I20260924 13:41:36.133538 12082 tablet_bootstrap.cc:492] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Bootstrap starting.
I20260924 13:41:36.133607 12081 tablet_bootstrap.cc:654] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:36.133940 12081 log.cc:824] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:36.134227 12082 tablet_bootstrap.cc:654] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:36.134543 12082 log.cc:824] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:36.134690 12081 tablet_bootstrap.cc:492] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: No bootstrap required, opened a new log
I20260924 13:41:36.134768 12081 ts_tablet_manager.cc:1397] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:41:36.134869 12083 tablet_bootstrap.cc:492] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f: Bootstrap starting.
I20260924 13:41:36.135195 12082 tablet_bootstrap.cc:492] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: No bootstrap required, opened a new log
I20260924 13:41:36.135245 12082 ts_tablet_manager.cc:1397] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:41:36.136894 12081 raft_consensus.cc:348] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.137059 12081 raft_consensus.cc:374] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:36.137107 12081 raft_consensus.cc:729] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6328c47ebb5c4a1e98f950658faaf1aa, State: Initialized, Role: FOLLOWER
I20260924 13:41:36.137012 12082 raft_consensus.cc:348] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.137174 12082 raft_consensus.cc:374] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:36.137220 12082 raft_consensus.cc:729] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f511b6a3b77a48ce86990e5eacb3c61e, State: Initialized, Role: FOLLOWER
I20260924 13:41:36.137226 12081 consensus_queue.cc:260] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.137348 12082 consensus_queue.cc:260] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.137511 12067 heartbeater.cc:503] Master 127.25.240.126:35575 was elected leader, sending a full tablet report...
I20260924 13:41:36.137650 12082 ts_tablet_manager.cc:1428] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 13:41:36.137676 11936 heartbeater.cc:503] Master 127.25.240.126:35575 was elected leader, sending a full tablet report...
I20260924 13:41:36.137827 12081 ts_tablet_manager.cc:1428] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 13:41:36.138327 12083 tablet_bootstrap.cc:654] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:36.138656 12083 log.cc:824] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:36.139305 12083 tablet_bootstrap.cc:492] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f: No bootstrap required, opened a new log
I20260924 13:41:36.139393 12083 ts_tablet_manager.cc:1397] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 13:41:36.141208 12083 raft_consensus.cc:348] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.141355 12083 raft_consensus.cc:374] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:36.141402 12083 raft_consensus.cc:729] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef987a2ef1f4c8780b791c490086c1f, State: Initialized, Role: FOLLOWER
I20260924 13:41:36.141515 12083 consensus_queue.cc:260] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.141788 12083 ts_tablet_manager.cc:1428] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 13:41:36.141831 11805 heartbeater.cc:503] Master 127.25.240.126:35575 was elected leader, sending a full tablet report...
W20260924 13:41:36.222314 11937 tablet.cc:2369] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:41:36.261085 12089 raft_consensus.cc:482] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:36.261207 12089 raft_consensus.cc:504] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.261695 12089 leader_election.cc:288] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f511b6a3b77a48ce86990e5eacb3c61e (127.25.240.66:39293), 6328c47ebb5c4a1e98f950658faaf1aa (127.25.240.67:44201)
I20260924 13:41:36.264422 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.261884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:39293 (local address 127.25.240.65:55861)
0924 13:41:36.261936 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:36.261949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:36.261993 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:36.262165 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:36.262166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:36.262167 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:36.262239 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.262241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.262867 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.262868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.263621 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.263623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.264166 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.264168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.264235 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.264239 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:36.264309 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:41:36.264350 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:41:36.264422 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.261982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:55861 (local address 127.25.240.66:39293)
0924 13:41:36.262037 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:36.262039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:36.262047 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:41:36.262076 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:36.262077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:36.262087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:36.262160 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:36.262313 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.262789 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.263716 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.264082 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.264156 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.264164 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:36.264316 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 13:41:36.264354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:41:36.264593 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.261972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:50781 (local address 127.25.240.67:44201)
0924 13:41:36.262028 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:36.262031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:36.262171 (+ 140us) server_negotiation.cc:407] Connection header received
0924 13:41:36.262262 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:36.262263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:36.262272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:36.262349 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:36.262510 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.262961 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.263852 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.264203 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.264291 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.264298 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:36.264461 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:41:36.264502 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 13:41:36.264770 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.261888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:44201 (local address 127.25.240.65:50781)
0924 13:41:36.262100 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:36.262109 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:36.262187 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:36.262355 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:36.262356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:36.262357 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:36.262437 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.262439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.263036 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.263037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.263767 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:36.263768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.264294 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.264295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:36.264379 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.264382 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:36.264452 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:41:36.264621 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":56,"thread_start_us":91,"threads_started":1}
I20260924 13:41:36.264752 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6de12c846a4056a733baeb92463dc5" candidate_uuid: "fef987a2ef1f4c8780b791c490086c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" is_pre_election: true
I20260924 13:41:36.264921 11890 raft_consensus.cc:2454] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef987a2ef1f4c8780b791c490086c1f in term 0.
I20260924 13:41:36.265096 12021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6de12c846a4056a733baeb92463dc5" candidate_uuid: "fef987a2ef1f4c8780b791c490086c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" is_pre_election: true
I20260924 13:41:36.265161 11695 leader_election.cc:302] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f511b6a3b77a48ce86990e5eacb3c61e, fef987a2ef1f4c8780b791c490086c1f; no voters:
I20260924 13:41:36.265247 12021 raft_consensus.cc:2454] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef987a2ef1f4c8780b791c490086c1f in term 0.
I20260924 13:41:36.265298 12089 raft_consensus.cc:2781] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:41:36.265345 12089 raft_consensus.cc:482] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:41:36.265375 12089 raft_consensus.cc:3037] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:36.266340 12089 raft_consensus.cc:504] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.266496 12089 leader_election.cc:288] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [CANDIDATE]: Term 1 election: Requested vote from peers f511b6a3b77a48ce86990e5eacb3c61e (127.25.240.66:39293), 6328c47ebb5c4a1e98f950658faaf1aa (127.25.240.67:44201)
I20260924 13:41:36.266636 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6de12c846a4056a733baeb92463dc5" candidate_uuid: "fef987a2ef1f4c8780b791c490086c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f511b6a3b77a48ce86990e5eacb3c61e"
I20260924 13:41:36.266638 12021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d6de12c846a4056a733baeb92463dc5" candidate_uuid: "fef987a2ef1f4c8780b791c490086c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6328c47ebb5c4a1e98f950658faaf1aa"
I20260924 13:41:36.266726 11890 raft_consensus.cc:3037] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:36.266722 12021 raft_consensus.cc:3037] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:36.267539 11890 raft_consensus.cc:2454] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef987a2ef1f4c8780b791c490086c1f in term 1.
I20260924 13:41:36.267577 12021 raft_consensus.cc:2454] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef987a2ef1f4c8780b791c490086c1f in term 1.
I20260924 13:41:36.267709 11695 leader_election.cc:302] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f511b6a3b77a48ce86990e5eacb3c61e, fef987a2ef1f4c8780b791c490086c1f; no voters:
I20260924 13:41:36.267794 12089 raft_consensus.cc:2781] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:36.267930 12089 raft_consensus.cc:686] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 1 LEADER]: Becoming Leader. State: Replica: fef987a2ef1f4c8780b791c490086c1f, State: Running, Role: LEADER
I20260924 13:41:36.268028 12089 consensus_queue.cc:237] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } }
I20260924 13:41:36.268814 11616 catalog_manager.cc:5495] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f reported cstate change: term changed from 0 to 1, leader changed from <none> to fef987a2ef1f4c8780b791c490086c1f (127.25.240.65). New cstate: current_term: 1 leader_uuid: "fef987a2ef1f4c8780b791c490086c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef987a2ef1f4c8780b791c490086c1f" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 32787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 } health_report { overall_health: UNKNOWN } } }
I20260924 13:41:36.307976 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.305744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46832 (local address 127.25.240.65:32787)
0924 13:41:36.305801 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:36.305804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:36.305875 (+ 71us) server_negotiation.cc:407] Connection header received
0924 13:41:36.305936 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:36.305936 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:36.305956 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:36.306027 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:41:36.306146 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.306611 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.307340 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:36.307525 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:36.307577 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:36.307701 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:41:36.307798 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:41:36.307855 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:36.307883 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 13:41:36.307898 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:41:36.309844 11890 raft_consensus.cc:1261] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 1 FOLLOWER]: Refusing update from remote peer fef987a2ef1f4c8780b791c490086c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:36.310206 12021 raft_consensus.cc:1261] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 1 FOLLOWER]: Refusing update from remote peer fef987a2ef1f4c8780b791c490086c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:36.310230 12089 consensus_queue.cc:1035] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f511b6a3b77a48ce86990e5eacb3c61e" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:36.310497 12089 consensus_queue.cc:1035] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6328c47ebb5c4a1e98f950658faaf1aa" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:36.314874 11616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:36.314975 11616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:36.315716 11616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 123171e410a8488aaed21e2fbf47ce10: Sending DeleteTablet for 3 replicas of tablet 5d6de12c846a4056a733baeb92463dc5
I20260924 13:41:36.316011 11870 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6de12c846a4056a733baeb92463dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:36 UTC) from {username='slave'} at 127.0.0.1:41844
I20260924 13:41:36.316037 12001 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6de12c846a4056a733baeb92463dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:36 UTC) from {username='slave'} at 127.0.0.1:43524
I20260924 13:41:36.316128 11739 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d6de12c846a4056a733baeb92463dc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:36 UTC) from {username='slave'} at 127.0.0.1:46820
I20260924 13:41:36.316311 12106 tablet_replica.cc:330] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f: stopping tablet replica
I20260924 13:41:36.316361 12105 tablet_replica.cc:330] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: stopping tablet replica
I20260924 13:41:36.316401 12106 raft_consensus.cc:2229] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:41:36.316460 12105 raft_consensus.cc:2229] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:36.316529 12106 raft_consensus.cc:2258] T 5d6de12c846a4056a733baeb92463dc5 P fef987a2ef1f4c8780b791c490086c1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:36.316557 12105 raft_consensus.cc:2258] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:36.316637 12104 tablet_replica.cc:330] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: stopping tablet replica
I20260924 13:41:36.316725 12104 raft_consensus.cc:2229] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:36.316807 12104 raft_consensus.cc:2258] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:36.316859 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 11677
I20260924 13:41:36.317989 12105 ts_tablet_manager.cc:1905] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:36.318862 12104 ts_tablet_manager.cc:1905] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:36.320089 12105 ts_tablet_manager.cc:1918] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:36.320164 12105 log.cc:1196] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/5d6de12c846a4056a733baeb92463dc5
I20260924 13:41:36.320451 12104 ts_tablet_manager.cc:1918] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:36.320446 12105 ts_tablet_manager.cc:1939] T 5d6de12c846a4056a733baeb92463dc5 P 6328c47ebb5c4a1e98f950658faaf1aa: Deleting consensus metadata
I20260924 13:41:36.320525 12104 log.cc:1196] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/5d6de12c846a4056a733baeb92463dc5
I20260924 13:41:36.320758 12104 ts_tablet_manager.cc:1939] T 5d6de12c846a4056a733baeb92463dc5 P f511b6a3b77a48ce86990e5eacb3c61e: Deleting consensus metadata
I20260924 13:41:36.320859 11600 catalog_manager.cc:4841] TS 6328c47ebb5c4a1e98f950658faaf1aa (127.25.240.67:44201): tablet 5d6de12c846a4056a733baeb92463dc5 (table test-table [id=9de8c515a7044509b51fcfabb0b55c1a]) successfully deleted
I20260924 13:41:36.321301 11603 catalog_manager.cc:4841] TS f511b6a3b77a48ce86990e5eacb3c61e (127.25.240.66:39293): tablet 5d6de12c846a4056a733baeb92463dc5 (table test-table [id=9de8c515a7044509b51fcfabb0b55c1a]) successfully deleted
W20260924 13:41:36.322671 11602 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:32787 (error 108)
I20260924 13:41:36.322862 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 11808
I20260924 13:41:36.323716 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:36.323593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:32787 (local address 127.0.0.1:46844)
0924 13:41:36.323642 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:36.323681 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:32787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:41:36.323812 11602 catalog_manager.cc:4584] TS fef987a2ef1f4c8780b791c490086c1f (127.25.240.65:32787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d6de12c846a4056a733baeb92463dc5: Network error: Client connection negotiation failed: client connection to 127.25.240.65:32787: connect: Connection refused (error 111)
I20260924 13:41:36.328023 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 11939
I20260924 13:41:36.332824 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 11585
2026-09-24T13:41:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (782 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:36 dist-test-slave-m5z1 krb5kdc[12110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:36 dist-test-slave-m5z1 krb5kdc[12110](info): set up 2 sockets
Sep 24 13:41:36 dist-test-slave-m5z1 krb5kdc[12110](info): commencing operation
krb5kdc: starting...
W20260924 13:41:37.393640 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:37 dist-test-slave-m5z1 krb5kdc[12110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:41:37.510295 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35925
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:35925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:37.591869 12123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:37.592064 12123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:37.592087 12123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:37.593353 12123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:37.593412 12123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:37.593436 12123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:37.593449 12123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:37.593461 12123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:37.594992 12123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:35925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:35925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:37.595239 12123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:37.595465 12123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:37.600119 12131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:37.600157 12129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:37.600260 12123 server_base.cc:1034] running on GCE node
W20260924 13:41:37.600394 12128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:37.600610 12123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:37.600934 12123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:37.602103 12123 hybrid_clock.cc:648] HybridClock initialized: now 1790257297602082 us; error 36 us; skew 500 ppm
Sep 24 13:41:37 dist-test-slave-m5z1 krb5kdc[12110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257297, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:37.604504 12123 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:41:37.604877 12123 webserver.cc:460] Webserver started at http://127.25.240.126:42427/ using document root <none> and password file <none>
I20260924 13:41:37.605059 12123 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:37.605098 12123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:37.605182 12123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:37.606223 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "fe93cb75342619ce5aa41121c9f17d8d"
server_key_iv: "0898d055803dc2dc72a4f66eff063ce6"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.606521 12123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "fe93cb75342619ce5aa41121c9f17d8d"
server_key_iv: "0898d055803dc2dc72a4f66eff063ce6"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.607823 12123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:41:37.608491 12138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:37.608667 12123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:37.608752 12123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "fe93cb75342619ce5aa41121c9f17d8d"
server_key_iv: "0898d055803dc2dc72a4f66eff063ce6"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.608822 12123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:37.623363 12123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:37.625176 12123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:37.625281 12123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:37.628983 12123 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:35925
I20260924 13:41:37.629026 12190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:35925 every 8 connection(s)
I20260924 13:41:37.629354 12123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:37.630259 12191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:37.632381 12191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1: Bootstrap starting.
I20260924 13:41:37.632932 12191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:37.633205 12191 log.cc:824] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:37.633935 12191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1: No bootstrap required, opened a new log
I20260924 13:41:37.634732 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 12123
I20260924 13:41:37.634827 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:37.635038 26561 external_mini_cluster.cc:1428] Setting key d4b9e15f1e0c33e4708e3b0be3db57a7
I20260924 13:41:37.635962 12191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35925 } }
I20260924 13:41:37.636129 12191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:37.636209 12191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1e4dd4786e4d34b176ccfdfd2d0fa1, State: Initialized, Role: FOLLOWER
I20260924 13:41:37.636340 12191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35925 } }
I20260924 13:41:37.636440 12191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:37.636485 12191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:37.636535 12191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:37.637584 12191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35925 } }
I20260924 13:41:37.637689 12191 leader_election.cc:302] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f1e4dd4786e4d34b176ccfdfd2d0fa1; no voters:
I20260924 13:41:37.637835 12191 leader_election.cc:288] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:37.637903 12196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:37.638070 12191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:37.638103 12196 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [term 1 LEADER]: Becoming Leader. State: Replica: 6f1e4dd4786e4d34b176ccfdfd2d0fa1, State: Running, Role: LEADER
I20260924 13:41:37.638302 12196 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35925 } }
I20260924 13:41:37.638717 12197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35925 } } }
I20260924 13:41:37.638792 12197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:37.639014 12198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f1e4dd4786e4d34b176ccfdfd2d0fa1. Latest consensus state: current_term: 1 leader_uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1e4dd4786e4d34b176ccfdfd2d0fa1" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 35925 } } }
I20260924 13:41:37.639313 12198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1 [sys.catalog]: This master's current role is: LEADER
W20260924 13:41:37.639909 12212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:41:37.640329 12212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:41:37.640398 12205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:37.640563 12205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:37.642163 12205 catalog_manager.cc:1347] Generated new cluster ID: 86d610299d1f49c991f8b52de9287132
I20260924 13:41:37.642213 12205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:41:37 dist-test-slave-m5z1 krb5kdc[12110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:37.646431 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:37.635536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.25.240.126:35925)
0924 13:41:37.635759 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:37.635768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:37.635803 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:41:37.635916 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:37.635923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:37.635977 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:37.636116 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:37.636899 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.638325 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.639430 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.639988 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.640101 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:37.642780 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:37.642783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:37.642784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:37.642795 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:37.644658 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:37.645016 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:37.645019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:37.645019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:37.645128 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:37.645306 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:37.645307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:37.645307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:37.645607 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:37.645739 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:37.646117 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 13:41:37.646242 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":100,"threads_started":1}
I20260924 13:41:37.663147 12205 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:37.663759 12205 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:37.672309 12205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1: Generated new TSK 0
I20260924 13:41:37.672511 12205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:41:37.708352 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35925
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:37.787981 12217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:37.788219 12217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:37.788259 12217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:37.789381 12217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:37.789443 12217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:37.789487 12217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:41:37.790930 12217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:37.791194 12217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:37.791410 12217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:37.796365 12223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:37.796376 12225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:37.796521 12217 server_base.cc:1034] running on GCE node
W20260924 13:41:37.796475 12222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:37.796820 12217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:37.797058 12217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:37.798211 12217 hybrid_clock.cc:648] HybridClock initialized: now 1790257297798206 us; error 35 us; skew 500 ppm
Sep 24 13:41:37 dist-test-slave-m5z1 krb5kdc[12110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257297, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:37.800560 12217 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:41:37.800982 12217 webserver.cc:460] Webserver started at http://127.25.240.65:42293/ using document root <none> and password file <none>
I20260924 13:41:37.801205 12217 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:37.801268 12217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:37.801421 12217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:37.802507 12217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "df0b98d4ed2342f4a80ed3db30804d58"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "13fbef9d325e93a511efaced322151a2"
server_key_iv: "65c1b9cc5015867a028af93dc68bd909"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.802865 12217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "df0b98d4ed2342f4a80ed3db30804d58"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "13fbef9d325e93a511efaced322151a2"
server_key_iv: "65c1b9cc5015867a028af93dc68bd909"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.804133 12217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:41:37.804862 12232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:37.805042 12217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:37.805148 12217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "df0b98d4ed2342f4a80ed3db30804d58"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "13fbef9d325e93a511efaced322151a2"
server_key_iv: "65c1b9cc5015867a028af93dc68bd909"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.805222 12217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:37.812865 12217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:37.814633 12217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:37.814785 12217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:37.815001 12217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:37.815285 12217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:37.815333 12217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:37.815378 12217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:37.815408 12217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:37.820923 12217 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:42463
I20260924 13:41:37.821007 12345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:42463 every 8 connection(s)
I20260924 13:41:37.821300 12217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:41:37.822788 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 12217
I20260924 13:41:37.822906 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:41:37.823104 26561 external_mini_cluster.cc:1428] Setting key 39d1c5b71874b98f3bc586c7180b7b88
Sep 24 13:41:37 dist-test-slave-m5z1 krb5kdc[12110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257297, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
I20260924 13:41:37.830973 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:37.822049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:33523 (local address 127.25.240.126:35925)
0924 13:41:37.822101 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:37.822104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:37.822322 (+ 218us) server_negotiation.cc:407] Connection header received
0924 13:41:37.822790 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:37.822791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:37.822804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:37.822877 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:37.823833 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.824600 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.825676 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.825916 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.826003 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:37.828090 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:37.828092 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:37.828092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:37.828102 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:37.829508 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:37.829962 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:37.829963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:37.829963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:37.830022 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:37.830276 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:37.830277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:37.830277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:37.830394 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:37.830463 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:37.830776 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 13:41:37.830882 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:41:37.831135 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:37.822037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35925 (local address 127.25.240.65:33523)
0924 13:41:37.822209 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:37.822243 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:37.822699 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:37.822984 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:37.822987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:37.823564 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:37.823749 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:37.823753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.824708 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.824709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:37.825510 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:37.825513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.826016 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.826017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:37.826149 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:37.826666 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:37.826674 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:37.827996 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:37.829647 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:37.829653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:37.829656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:37.829879 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:37.830085 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:37.830086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:37.830087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:37.830203 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:37.830464 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:37.830471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:37.830633 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 13:41:37.830743 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 13:41:37.830906 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
I20260924 13:41:37.831635 12346 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35925
I20260924 13:41:37.831745 12346 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:37.831959 12346 heartbeater.cc:511] Master 127.25.240.126:35925 requested a full tablet report, sending...
I20260924 13:41:37.832463 12155 ts_manager.cc:194] Registered new tserver with Master: df0b98d4ed2342f4a80ed3db30804d58 (127.25.240.65:42463)
I20260924 13:41:37.833122 12155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:33523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:41:37.848935 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35925
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:37.927963 12351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:37.928186 12351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:37.928225 12351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:37.929344 12351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:37.929411 12351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:37.929458 12351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:41:37.930897 12351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:37.931160 12351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:37.931370 12351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:37.936460 12357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:37.936442 12356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:37.936632 12351 server_base.cc:1034] running on GCE node
W20260924 13:41:37.936443 12359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:37.936882 12351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:37.937119 12351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:37.938272 12351 hybrid_clock.cc:648] HybridClock initialized: now 1790257297938263 us; error 34 us; skew 500 ppm
Sep 24 13:41:37 dist-test-slave-m5z1 krb5kdc[12110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257297, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:37.940620 12351 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:41:37.941027 12351 webserver.cc:460] Webserver started at http://127.25.240.66:43379/ using document root <none> and password file <none>
I20260924 13:41:37.941208 12351 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:37.941248 12351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:37.941331 12351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:37.942302 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "83d9d77fc46a41019f0b5edf5845fc83"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "1c4c597b9f9721d564084ea0dad2be1d"
server_key_iv: "a4ccfa0359b768e2309bfe9f6160e71c"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.942595 12351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "83d9d77fc46a41019f0b5edf5845fc83"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "1c4c597b9f9721d564084ea0dad2be1d"
server_key_iv: "a4ccfa0359b768e2309bfe9f6160e71c"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.943899 12351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:41:37.944707 12366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:37.944926 12351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:41:37.945010 12351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "83d9d77fc46a41019f0b5edf5845fc83"
format_stamp: "Formatted at 2026-09-24 13:41:37 on dist-test-slave-m5z1"
server_key: "1c4c597b9f9721d564084ea0dad2be1d"
server_key_iv: "a4ccfa0359b768e2309bfe9f6160e71c"
server_key_version: "encryptionkey@0"
I20260924 13:41:37.945065 12351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:37.955399 12351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:37.957474 12351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:37.957641 12351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:37.957811 12351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:37.958065 12351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:37.958096 12351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:37.958115 12351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:37.958128 12351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:37.963622 12351 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:44433
I20260924 13:41:37.963691 12479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:44433 every 8 connection(s)
I20260924 13:41:37.963981 12351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
Sep 24 13:41:37 dist-test-slave-m5z1 krb5kdc[12110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257297, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:37.972683 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:37.964655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:42613 (local address 127.25.240.126:35925)
0924 13:41:37.964712 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:37.964714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:37.964898 (+ 184us) server_negotiation.cc:407] Connection header received
0924 13:41:37.965364 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:37.965365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:37.965374 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:37.965419 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:37.966384 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.966934 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.967941 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.968132 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.968196 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:37.970208 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:37.970209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:37.970209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:37.970215 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:37.971563 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:37.971944 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:37.971945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:37.971945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:37.971995 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:37.972196 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:37.972196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:37.972197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:37.972291 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:37.972338 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:37.972561 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 13:41:37.972622 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:41:37.972851 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:37.964614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35925 (local address 127.25.240.66:42613)
0924 13:41:37.964780 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:37.964821 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:37.965268 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:37.965482 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:37.965485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:37.966091 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:37.966271 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:37.966276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.967025 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.967026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:37.967802 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:37.967804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:37.968209 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:37.968210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:37.968313 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:37.968740 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:37.968747 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:37.970120 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:37.971646 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:37.971652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:37.971654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:37.971864 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:37.972048 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:37.972049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:37.972049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:37.972146 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:37.972340 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:37.972344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:37.972458 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 13:41:37.972539 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 13:41:37.972653 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260924 13:41:37.973510 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 12351
I20260924 13:41:37.973517 12480 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35925
I20260924 13:41:37.973641 12480 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:37.973649 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:41:37.973846 12480 heartbeater.cc:511] Master 127.25.240.126:35925 requested a full tablet report, sending...
I20260924 13:41:37.973870 26561 external_mini_cluster.cc:1428] Setting key 36667351b5bd0bff4e22648af0f89437
I20260924 13:41:37.974269 12154 ts_manager.cc:194] Registered new tserver with Master: 83d9d77fc46a41019f0b5edf5845fc83 (127.25.240.66:44433)
I20260924 13:41:37.974711 12154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:42613
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:41:37.998651 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35925
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:38.078293 12485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:38.078505 12485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:38.078528 12485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:38.079694 12485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:38.079734 12485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:38.079761 12485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:41:38.081194 12485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:35925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:38.081445 12485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:38.081681 12485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:38.086598 12491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:38.086782 12485 server_base.cc:1034] running on GCE node
W20260924 13:41:38.086632 12493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:38.086611 12490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:38.087054 12485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:38.087302 12485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:38.088454 12485 hybrid_clock.cc:648] HybridClock initialized: now 1790257298088443 us; error 25 us; skew 500 ppm
Sep 24 13:41:38 dist-test-slave-m5z1 krb5kdc[12110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257298, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:38.090847 12485 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:41:38.091275 12485 webserver.cc:460] Webserver started at http://127.25.240.67:35431/ using document root <none> and password file <none>
I20260924 13:41:38.091488 12485 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:38.091553 12485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:38.091693 12485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:38.092710 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "dd8ce598327746a68003e280335e3068"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "8bd3576fb751140a947458fef21fc673"
server_key_iv: "b951e844daac890ef4d04df2440830d8"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.093060 12485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "dd8ce598327746a68003e280335e3068"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "8bd3576fb751140a947458fef21fc673"
server_key_iv: "b951e844daac890ef4d04df2440830d8"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.094401 12485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:41:38.095103 12500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.095260 12485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.095350 12485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "dd8ce598327746a68003e280335e3068"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "8bd3576fb751140a947458fef21fc673"
server_key_iv: "b951e844daac890ef4d04df2440830d8"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.095423 12485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:38.110628 12485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:38.112602 12485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:38.112754 12485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:38.112987 12485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:38.113277 12485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:38.113332 12485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.113371 12485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:38.113395 12485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.118979 12485 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:34937
I20260924 13:41:38.119032 12613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:34937 every 8 connection(s)
I20260924 13:41:38.119323 12485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:41:38.122797 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 12485
I20260924 13:41:38.122913 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:41:38.123103 26561 external_mini_cluster.cc:1428] Setting key a1f97d459d7b3e20be5e72d4d835ec59
Sep 24 13:41:38 dist-test-slave-m5z1 krb5kdc[12110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257298, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:38.128616 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.119953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:35609 (local address 127.25.240.126:35925)
0924 13:41:38.120013 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.120015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.120240 (+ 225us) server_negotiation.cc:407] Connection header received
0924 13:41:38.120690 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.120691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.120701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.120752 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:38.121817 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.122555 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.123773 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.124067 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.124118 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.126097 (+ 1979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:38.126098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:38.126098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:38.126104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:38.127480 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:38.127808 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:38.127808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:38.127809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:38.127856 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:38.128031 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:38.128032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:38.128032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:38.128129 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:38.128180 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.128487 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.128554 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 13:41:38.128912 12616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.119959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:35925 (local address 127.25.240.67:35609)
0924 13:41:38.120129 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.120163 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.120610 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.120800 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.120804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.121410 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:38.121723 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.121729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.122643 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.122644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.123701 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.123704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.124125 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.124125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.124224 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.124625 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:38.124633 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:38.125999 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:38.127546 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:38.127551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:38.127553 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:38.127747 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:38.127900 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:38.127901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:38.127901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:38.127983 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:38.128222 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:38.128229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:38.128358 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.128465 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.128591 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260924 13:41:38.129400 12614 heartbeater.cc:347] Connected to a master server at 127.25.240.126:35925
I20260924 13:41:38.129490 12614 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:38.129685 12614 heartbeater.cc:511] Master 127.25.240.126:35925 requested a full tablet report, sending...
I20260924 13:41:38.130014 12154 ts_manager.cc:194] Registered new tserver with Master: dd8ce598327746a68003e280335e3068 (127.25.240.67:34937)
I20260924 13:41:38.130370 12154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:35609
I20260924 13:41:38.134739 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:41:38.142602 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.136206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.25.240.126:35925)
0924 13:41:38.136288 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.136290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.136396 (+ 106us) server_negotiation.cc:407] Connection header received
0924 13:41:38.136484 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.136485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.136496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.136545 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:38.137179 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.137852 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.138662 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.138984 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.139058 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.140397 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:38.140398 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:38.140398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:38.140404 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:38.141534 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:38.141937 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:38.141938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:38.141939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:38.141986 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:38.142161 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:38.142161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:38.142161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:38.142270 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:38.142353 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.142451 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.142496 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 13:41:38.144203 12154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:41:38.145737 12154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:41:38.151959 12624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.148010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:42463 (local address 127.0.0.1:57546)
0924 13:41:38.148099 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.148109 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.148162 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.148728 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.148729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.148732 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:38.148782 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.148783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.150042 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.150043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.151097 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.151098 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.151723 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.151724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.151784 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.151788 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.151832 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.151872 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260924 13:41:38.151971 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.148370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.25.240.65:42463)
0924 13:41:38.148558 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.148562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.148572 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:41:38.148611 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.148613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.148627 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.148848 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:38.148915 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.149947 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.151171 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.151638 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.151719 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.151820 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.151849 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.151886 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":82,"threads_started":1}
I20260924 13:41:38.152032 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.147864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.25.240.67:34937)
0924 13:41:38.148030 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.148035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.148044 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:41:38.148083 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.148085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.148101 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.148185 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:38.148437 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.149596 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.150746 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.151440 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.151514 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.151566 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.151915 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.151960 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260924 13:41:38.152078 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.148162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:44433 (local address 127.0.0.1:56096)
0924 13:41:38.148334 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.148343 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.148393 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.148710 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.148711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.148713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:38.148760 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.148762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.149650 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.149651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.150930 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.150933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.151865 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.151866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.151922 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.151925 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.151973 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.152007 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":48,"thread_start_us":43,"threads_started":1}
I20260924 13:41:38.152236 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.147784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34937 (local address 127.0.0.1:33580)
0924 13:41:38.147885 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.147922 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.147984 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.148261 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.148264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.148267 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:38.148348 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.148351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.149701 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.149702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.150661 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.150664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.151600 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.151601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.151694 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.151708 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.151777 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.151826 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":32,"thread_start_us":58,"threads_started":1}
I20260924 13:41:38.152263 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.148276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.25.240.66:44433)
0924 13:41:38.148472 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.148476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.148486 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:41:38.148522 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.148524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.148538 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.148626 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:38.148839 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.149531 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.151012 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.151724 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.151785 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.151833 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.152157 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.152207 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260924 13:41:38.152971 12414 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf42a8225344290aa74614cd317e580 (DEFAULT_TABLE table=test-table [id=62845ebf2723441b94393965d22655bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:38.153003 12548 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf42a8225344290aa74614cd317e580 (DEFAULT_TABLE table=test-table [id=62845ebf2723441b94393965d22655bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:38.153334 12548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf42a8225344290aa74614cd317e580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:38.153339 12414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf42a8225344290aa74614cd317e580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:38.153980 12280 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf42a8225344290aa74614cd317e580 (DEFAULT_TABLE table=test-table [id=62845ebf2723441b94393965d22655bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:38.154264 12280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf42a8225344290aa74614cd317e580. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:38.155714 12628 tablet_bootstrap.cc:492] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Bootstrap starting.
I20260924 13:41:38.156493 12629 tablet_bootstrap.cc:492] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: Bootstrap starting.
I20260924 13:41:38.156546 12628 tablet_bootstrap.cc:654] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:38.157117 12628 log.cc:824] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:38.157176 12629 tablet_bootstrap.cc:654] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:38.157235 12630 tablet_bootstrap.cc:492] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58: Bootstrap starting.
I20260924 13:41:38.157699 12629 log.cc:824] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:38.158072 12630 tablet_bootstrap.cc:654] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:38.158126 12628 tablet_bootstrap.cc:492] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: No bootstrap required, opened a new log
I20260924 13:41:38.158206 12628 ts_tablet_manager.cc:1397] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 13:41:38.158561 12630 log.cc:824] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:38.159173 12630 tablet_bootstrap.cc:492] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58: No bootstrap required, opened a new log
I20260924 13:41:38.159250 12630 ts_tablet_manager.cc:1397] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:41:38.159766 12628 raft_consensus.cc:348] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.159901 12628 raft_consensus.cc:374] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:38.159924 12628 raft_consensus.cc:729] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d9d77fc46a41019f0b5edf5845fc83, State: Initialized, Role: FOLLOWER
I20260924 13:41:38.160040 12628 consensus_queue.cc:260] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.160306 12628 ts_tablet_manager.cc:1428] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:41:38.160346 12480 heartbeater.cc:503] Master 127.25.240.126:35925 was elected leader, sending a full tablet report...
I20260924 13:41:38.160935 12629 tablet_bootstrap.cc:492] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: No bootstrap required, opened a new log
I20260924 13:41:38.161002 12629 ts_tablet_manager.cc:1397] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 13:41:38.160928 12630 raft_consensus.cc:348] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.161098 12630 raft_consensus.cc:374] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:38.161139 12630 raft_consensus.cc:729] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0b98d4ed2342f4a80ed3db30804d58, State: Initialized, Role: FOLLOWER
I20260924 13:41:38.161278 12630 consensus_queue.cc:260] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.161540 12630 ts_tablet_manager.cc:1428] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:41:38.161584 12346 heartbeater.cc:503] Master 127.25.240.126:35925 was elected leader, sending a full tablet report...
I20260924 13:41:38.162704 12629 raft_consensus.cc:348] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.162824 12629 raft_consensus.cc:374] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:38.162845 12629 raft_consensus.cc:729] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8ce598327746a68003e280335e3068, State: Initialized, Role: FOLLOWER
I20260924 13:41:38.162971 12629 consensus_queue.cc:260] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.163228 12629 ts_tablet_manager.cc:1428] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 13:41:38.163252 12614 heartbeater.cc:503] Master 127.25.240.126:35925 was elected leader, sending a full tablet report...
W20260924 13:41:38.214536 12481 tablet.cc:2369] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:41:38.308816 12635 raft_consensus.cc:482] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:38.308939 12635 raft_consensus.cc:504] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.309314 12635 leader_election.cc:288] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8ce598327746a68003e280335e3068 (127.25.240.67:34937), 83d9d77fc46a41019f0b5edf5845fc83 (127.25.240.66:44433)
I20260924 13:41:38.312561 12348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.309543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34937 (local address 127.25.240.65:55063)
0924 13:41:38.309622 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.309634 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.309680 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.309882 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.309883 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.309885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:38.309959 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.309961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.310886 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.310888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.311734 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.311739 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.312297 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.312299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.312383 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.312388 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.312443 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.312474 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 13:41:38.312564 12623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.309684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:55063 (local address 127.25.240.67:34937)
0924 13:41:38.309734 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.309736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.309746 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:41:38.309777 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.309778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.309788 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.309858 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:38.310026 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.310783 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.311814 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.312221 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.312280 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.312287 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.312449 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.312498 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:41:38.312636 12626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.309780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:56369 (local address 127.25.240.66:44433)
0924 13:41:38.309822 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.309824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.309855 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:41:38.309965 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.309966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.309973 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.310035 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:38.310152 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.310843 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.311862 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.312284 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.312354 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.312364 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.312523 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.312562 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:41:38.312695 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.309691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:44433 (local address 127.25.240.65:56369)
0924 13:41:38.309807 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.309816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.309865 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.310052 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.310053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.310054 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:38.310100 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.310101 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.310932 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.310933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.311765 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.311767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.312363 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.312364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.312445 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.312449 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.312506 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.312543 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":83,"threads_started":1}
I20260924 13:41:38.313268 12568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf42a8225344290aa74614cd317e580" candidate_uuid: "df0b98d4ed2342f4a80ed3db30804d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8ce598327746a68003e280335e3068" is_pre_election: true
I20260924 13:41:38.313444 12568 raft_consensus.cc:2454] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0b98d4ed2342f4a80ed3db30804d58 in term 0.
I20260924 13:41:38.313706 12236 leader_election.cc:302] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd8ce598327746a68003e280335e3068, df0b98d4ed2342f4a80ed3db30804d58; no voters:
I20260924 13:41:38.313692 12434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf42a8225344290aa74614cd317e580" candidate_uuid: "df0b98d4ed2342f4a80ed3db30804d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d9d77fc46a41019f0b5edf5845fc83" is_pre_election: true
I20260924 13:41:38.313827 12434 raft_consensus.cc:2454] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0b98d4ed2342f4a80ed3db30804d58 in term 0.
I20260924 13:41:38.313817 12635 raft_consensus.cc:2781] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:41:38.313869 12635 raft_consensus.cc:482] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:41:38.313894 12635 raft_consensus.cc:3037] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:38.314611 12635 raft_consensus.cc:504] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.314746 12635 leader_election.cc:288] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [CANDIDATE]: Term 1 election: Requested vote from peers dd8ce598327746a68003e280335e3068 (127.25.240.67:34937), 83d9d77fc46a41019f0b5edf5845fc83 (127.25.240.66:44433)
I20260924 13:41:38.314950 12434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf42a8225344290aa74614cd317e580" candidate_uuid: "df0b98d4ed2342f4a80ed3db30804d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d9d77fc46a41019f0b5edf5845fc83"
I20260924 13:41:38.314965 12568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf42a8225344290aa74614cd317e580" candidate_uuid: "df0b98d4ed2342f4a80ed3db30804d58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8ce598327746a68003e280335e3068"
I20260924 13:41:38.315037 12568 raft_consensus.cc:3037] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:38.315042 12434 raft_consensus.cc:3037] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:38.315724 12568 raft_consensus.cc:2454] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0b98d4ed2342f4a80ed3db30804d58 in term 1.
I20260924 13:41:38.315838 12434 raft_consensus.cc:2454] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0b98d4ed2342f4a80ed3db30804d58 in term 1.
I20260924 13:41:38.315871 12236 leader_election.cc:302] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd8ce598327746a68003e280335e3068, df0b98d4ed2342f4a80ed3db30804d58; no voters:
I20260924 13:41:38.315948 12635 raft_consensus.cc:2781] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:38.316100 12635 raft_consensus.cc:686] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [term 1 LEADER]: Becoming Leader. State: Replica: df0b98d4ed2342f4a80ed3db30804d58, State: Running, Role: LEADER
I20260924 13:41:38.316196 12635 consensus_queue.cc:237] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } }
I20260924 13:41:38.317056 12155 catalog_manager.cc:5495] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 reported cstate change: term changed from 0 to 1, leader changed from <none> to df0b98d4ed2342f4a80ed3db30804d58 (127.25.240.65). New cstate: current_term: 1 leader_uuid: "df0b98d4ed2342f4a80ed3db30804d58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df0b98d4ed2342f4a80ed3db30804d58" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 } health_report { overall_health: UNKNOWN } } }
W20260924 13:41:38.322005 12347 tablet.cc:2369] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:41:38.331769 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.329549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.25.240.65:42463)
0924 13:41:38.329614 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.329616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.329647 (+ 31us) server_negotiation.cc:407] Connection header received
0924 13:41:38.329728 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.329729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.329744 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.329803 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:41:38.329929 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.330371 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.331066 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.331241 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.331290 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.331426 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:41:38.331506 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:41:38.331576 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.331695 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.331705 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:41:38.333598 12434 raft_consensus.cc:1261] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 1 FOLLOWER]: Refusing update from remote peer df0b98d4ed2342f4a80ed3db30804d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:38.333684 12568 raft_consensus.cc:1261] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 1 FOLLOWER]: Refusing update from remote peer df0b98d4ed2342f4a80ed3db30804d58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:38.333948 12638 consensus_queue.cc:1035] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83d9d77fc46a41019f0b5edf5845fc83" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:38.334057 12635 consensus_queue.cc:1035] T 8bf42a8225344290aa74614cd317e580 P df0b98d4ed2342f4a80ed3db30804d58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8ce598327746a68003e280335e3068" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:38.336386 12641 mvcc.cc:205] Tried to move back new op lower bound from 7332893893971701760 to 7332893893903790080. Current Snapshot: MvccSnapshot[applied={T|T < 7332893893971701760}]
I20260924 13:41:38.337013 12643 mvcc.cc:205] Tried to move back new op lower bound from 7332893893971701760 to 7332893893903790080. Current Snapshot: MvccSnapshot[applied={T|T < 7332893893971701760}]
I20260924 13:41:38.337518 12642 mvcc.cc:205] Tried to move back new op lower bound from 7332893893971701760 to 7332893893903790080. Current Snapshot: MvccSnapshot[applied={T|T < 7332893893971701760 or (T in {7332893893971701760})}]
I20260924 13:41:38.338564 12155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:38.338656 12155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:38.339412 12155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f1e4dd4786e4d34b176ccfdfd2d0fa1: Sending DeleteTablet for 3 replicas of tablet 8bf42a8225344290aa74614cd317e580
I20260924 13:41:38.339733 12548 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf42a8225344290aa74614cd317e580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:38 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:33580
I20260924 13:41:38.339775 12280 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf42a8225344290aa74614cd317e580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:38 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:57546
I20260924 13:41:38.339839 12414 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf42a8225344290aa74614cd317e580 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:38 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:56096
I20260924 13:41:38.339987 12650 tablet_replica.cc:330] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: stopping tablet replica
I20260924 13:41:38.340037 12652 tablet_replica.cc:330] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: stopping tablet replica
I20260924 13:41:38.340128 12652 raft_consensus.cc:2229] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:38.340142 12650 raft_consensus.cc:2229] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:38.340232 12652 raft_consensus.cc:2258] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:38.340355 12650 raft_consensus.cc:2258] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:38.340487 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 12217
I20260924 13:41:38.340819 12650 ts_tablet_manager.cc:1905] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:38.342970 12650 ts_tablet_manager.cc:1918] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:38.343046 12650 log.cc:1196] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/8bf42a8225344290aa74614cd317e580
I20260924 13:41:38.343120 12652 ts_tablet_manager.cc:1905] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:38.343333 12650 ts_tablet_manager.cc:1939] T 8bf42a8225344290aa74614cd317e580 P dd8ce598327746a68003e280335e3068: Deleting consensus metadata
I20260924 13:41:38.343780 12142 catalog_manager.cc:4841] TS dd8ce598327746a68003e280335e3068 (127.25.240.67:34937): tablet 8bf42a8225344290aa74614cd317e580 (table test-table [id=62845ebf2723441b94393965d22655bd]) successfully deleted
I20260924 13:41:38.344635 12652 ts_tablet_manager.cc:1918] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:38.344691 12652 log.cc:1196] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/8bf42a8225344290aa74614cd317e580
I20260924 13:41:38.344944 12652 ts_tablet_manager.cc:1939] T 8bf42a8225344290aa74614cd317e580 P 83d9d77fc46a41019f0b5edf5845fc83: Deleting consensus metadata
I20260924 13:41:38.345291 12142 catalog_manager.cc:4841] TS 83d9d77fc46a41019f0b5edf5845fc83 (127.25.240.66:44433): tablet 8bf42a8225344290aa74614cd317e580 (table test-table [id=62845ebf2723441b94393965d22655bd]) successfully deleted
W20260924 13:41:38.346138 12142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:42463 (error 108)
I20260924 13:41:38.346329 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 12351
I20260924 13:41:38.347275 12622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.346515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:42463 (local address 127.0.0.1:57570)
0924 13:41:38.347209 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.347248 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:42463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":672}
W20260924 13:41:38.347396 12142 catalog_manager.cc:4584] TS df0b98d4ed2342f4a80ed3db30804d58 (127.25.240.65:42463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bf42a8225344290aa74614cd317e580: Network error: Client connection negotiation failed: client connection to 127.25.240.65:42463: connect: Connection refused (error 111)
I20260924 13:41:38.352169 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 12485
I20260924 13:41:38.357038 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 12123
2026-09-24T13:41:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2025 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T13:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:38Z Disabled control of system clock
I20260924 13:41:38.384328 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:33303
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:33303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:38.463692 12657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:38.463894 12657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:38.463917 12657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:38.465056 12657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:38.465098 12657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:38.465111 12657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:38.465123 12657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:38.465135 12657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:38.466527 12657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:33303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:33303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:38.466748 12657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:38.467041 12657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:38.471959 12665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:38.471966 12662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:38.471967 12663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:38.472200 12657 server_base.cc:1034] running on GCE node
I20260924 13:41:38.472438 12657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:38.472704 12657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:38.473861 12657 hybrid_clock.cc:648] HybridClock initialized: now 1790257298473843 us; error 39 us; skew 500 ppm
I20260924 13:41:38.475060 12657 webserver.cc:460] Webserver started at http://127.25.240.126:40425/ using document root <none> and password file <none>
I20260924 13:41:38.475303 12657 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:38.475385 12657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:38.475528 12657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:38.476581 12657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "32dc8896cadc4ceaa1a55ad343404510"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "8b34953032e35435cc18f0a0f16e0420"
server_key_iv: "a7405a5732709f5b89a68a7ea657f6de"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.476918 12657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "32dc8896cadc4ceaa1a55ad343404510"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "8b34953032e35435cc18f0a0f16e0420"
server_key_iv: "a7405a5732709f5b89a68a7ea657f6de"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.478224 12657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:38.478889 12671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.479051 12657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:41:38.479147 12657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "32dc8896cadc4ceaa1a55ad343404510"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "8b34953032e35435cc18f0a0f16e0420"
server_key_iv: "a7405a5732709f5b89a68a7ea657f6de"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.479216 12657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:38.500219 12657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:38.500551 12657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:38.500717 12657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:38.504834 12657 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:33303
I20260924 13:41:38.504858 12723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:33303 every 8 connection(s)
I20260924 13:41:38.505231 12657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:38.506213 12724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:38.508507 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 12657
I20260924 13:41:38.508625 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:38.508809 26561 external_mini_cluster.cc:1428] Setting key a11ebf1a18c97e1fe632da8adb442e0a
I20260924 13:41:38.508971 12724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510: Bootstrap starting.
I20260924 13:41:38.509686 12724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:38.510026 12724 log.cc:824] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:38.511150 12724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510: No bootstrap required, opened a new log
I20260924 13:41:38.513113 12724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc8896cadc4ceaa1a55ad343404510" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33303 } }
I20260924 13:41:38.513281 12724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:38.513322 12724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dc8896cadc4ceaa1a55ad343404510, State: Initialized, Role: FOLLOWER
I20260924 13:41:38.513464 12724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc8896cadc4ceaa1a55ad343404510" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33303 } }
I20260924 13:41:38.513528 12724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:38.513600 12724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:38.513651 12724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:38.514235 12724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc8896cadc4ceaa1a55ad343404510" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33303 } }
I20260924 13:41:38.514369 12724 leader_election.cc:302] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32dc8896cadc4ceaa1a55ad343404510; no voters:
I20260924 13:41:38.514549 12724 leader_election.cc:288] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:38.514698 12729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:38.514838 12724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:38.514865 12729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [term 1 LEADER]: Becoming Leader. State: Replica: 32dc8896cadc4ceaa1a55ad343404510, State: Running, Role: LEADER
I20260924 13:41:38.514961 12729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc8896cadc4ceaa1a55ad343404510" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33303 } }
I20260924 13:41:38.515357 12730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32dc8896cadc4ceaa1a55ad343404510" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc8896cadc4ceaa1a55ad343404510" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33303 } } }
I20260924 13:41:38.515440 12730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:38.515601 12731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32dc8896cadc4ceaa1a55ad343404510. Latest consensus state: current_term: 1 leader_uuid: "32dc8896cadc4ceaa1a55ad343404510" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc8896cadc4ceaa1a55ad343404510" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 33303 } } }
I20260924 13:41:38.515724 12731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:38.515772 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.509634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39944 (local address 127.25.240.126:33303)
0924 13:41:38.509975 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.509983 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.510016 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:41:38.510575 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.510584 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.510658 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.510811 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:38.511171 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.512148 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.513333 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.513945 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.514039 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.514865 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:38.514869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:38.514869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:38.515110 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:38.515152 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:38.515168 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:38.515208 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.515361 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.515452 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":74,"threads_started":1}
I20260924 13:41:38.515641 12737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:38.516525 12737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:38.517979 12737 catalog_manager.cc:1347] Generated new cluster ID: 583b94cb117842c498b33fba95a849e8
I20260924 13:41:38.518034 12737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:38.562381 12737 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:38.562963 12737 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:38.570581 12737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510: Generated new TSK 0
I20260924 13:41:38.570780 12737 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:38.574733 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33303
--builtin_ntp_servers=127.25.240.84:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:38.655241 12748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:38.655437 12748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:38.655459 12748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:38.656608 12748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:38.656648 12748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:38.656695 12748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:41:38.658043 12748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:38.658275 12748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:38.658551 12748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:38.663335 12753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:38.663370 12754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:38.663367 12756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:38.663465 12748 server_base.cc:1034] running on GCE node
I20260924 13:41:38.663674 12748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:38.663879 12748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:38.665059 12748 hybrid_clock.cc:648] HybridClock initialized: now 1790257298665047 us; error 25 us; skew 500 ppm
I20260924 13:41:38.666393 12748 webserver.cc:460] Webserver started at http://127.25.240.65:36337/ using document root <none> and password file <none>
I20260924 13:41:38.666620 12748 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:38.666667 12748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:38.666836 12748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:38.667857 12748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "304682c003de4b87b2f4fd26298d483c"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "5fd701a4bd51e6224e8e1e22edecefb2"
server_key_iv: "47aa58a317bb0d8c1e8ebeaddecee591"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.668215 12748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "304682c003de4b87b2f4fd26298d483c"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "5fd701a4bd51e6224e8e1e22edecefb2"
server_key_iv: "47aa58a317bb0d8c1e8ebeaddecee591"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.669525 12748 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:38.670341 12762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.670534 12748 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:38.670620 12748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "304682c003de4b87b2f4fd26298d483c"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "5fd701a4bd51e6224e8e1e22edecefb2"
server_key_iv: "47aa58a317bb0d8c1e8ebeaddecee591"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.670689 12748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:38.701740 12748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:38.702039 12748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:38.702184 12748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:38.702405 12748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:38.702718 12748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:38.702749 12748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.702797 12748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:38.702821 12748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.708309 12748 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:36421
I20260924 13:41:38.708371 12875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:36421 every 8 connection(s)
I20260924 13:41:38.708683 12748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:41:38.710011 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 12748
I20260924 13:41:38.710115 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:41:38.710300 26561 external_mini_cluster.cc:1428] Setting key 75fd2b8e977bcc0864a43408c7c6c598
I20260924 13:41:38.711330 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33303
--builtin_ntp_servers=127.25.240.84:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 13:41:38.714675 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.709358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:50357 (local address 127.25.240.126:33303)
0924 13:41:38.709413 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.709414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.709661 (+ 247us) server_negotiation.cc:407] Connection header received
0924 13:41:38.710353 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.710355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.710369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.710445 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:38.711303 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.711864 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.713173 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.713499 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.713588 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.714435 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:38.714436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:38.714437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:38.714441 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:38.714454 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:38.714458 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:38.714501 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.714596 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.714617 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:41:38.714982 12878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.709345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33303 (local address 127.25.240.65:50357)
0924 13:41:38.709531 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.709590 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.710236 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.710521 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.710530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.710539 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:38.711203 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.711210 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.711961 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.711962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.713082 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.713085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.713618 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.713620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.713774 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.714287 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:38.714287 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:38.714324 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:38.714352 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:38.714516 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:38.714521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:38.714524 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.714591 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.714701 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260924 13:41:38.715502 12876 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33303
I20260924 13:41:38.715602 12876 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:38.715808 12876 heartbeater.cc:511] Master 127.25.240.126:33303 requested a full tablet report, sending...
I20260924 13:41:38.716356 12688 ts_manager.cc:194] Registered new tserver with Master: 304682c003de4b87b2f4fd26298d483c (127.25.240.65:36421)
I20260924 13:41:38.717383 12688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.65:50357
W20260924 13:41:38.792493 12879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:38.792685 12879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:38.792706 12879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:38.793820 12879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:38.793856 12879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:38.793881 12879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:41:38.795140 12879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:38.795316 12879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:38.795539 12879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:38.800251 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:38.800438 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:38.800565 12879 server_base.cc:1034] running on GCE node
W20260924 13:41:38.800624 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:38.800822 12879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:38.801014 12879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:38.802162 12879 hybrid_clock.cc:648] HybridClock initialized: now 1790257298802152 us; error 38 us; skew 500 ppm
I20260924 13:41:38.810518 12879 webserver.cc:460] Webserver started at http://127.25.240.66:44731/ using document root <none> and password file <none>
I20260924 13:41:38.810765 12879 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:38.810830 12879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:38.811025 12879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:38.812047 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "7acbc99fdda947b5b27abc98932f2595"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "d997b8ce898fadca170486f4481c930f"
server_key_iv: "d49870c63d55dd915c4a3df7f2dbf3a5"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.812388 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "7acbc99fdda947b5b27abc98932f2595"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "d997b8ce898fadca170486f4481c930f"
server_key_iv: "d49870c63d55dd915c4a3df7f2dbf3a5"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.813690 12879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:38.814476 12893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.814646 12879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:38.814738 12879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "7acbc99fdda947b5b27abc98932f2595"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "d997b8ce898fadca170486f4481c930f"
server_key_iv: "d49870c63d55dd915c4a3df7f2dbf3a5"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.814826 12879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:38.831557 12879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:38.831846 12879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:38.831990 12879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:38.832193 12879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:38.832496 12879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:38.832551 12879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.832592 12879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:38.832619 12879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.838390 12879 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:32785
I20260924 13:41:38.838430 13006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:32785 every 8 connection(s)
I20260924 13:41:38.838723 12879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:41:38.843652 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.839371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:47123 (local address 127.25.240.126:33303)
0924 13:41:38.839447 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.839449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.839640 (+ 191us) server_negotiation.cc:407] Connection header received
0924 13:41:38.840142 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.840144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.840155 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.840212 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:38.840822 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.841351 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.842392 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.842577 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.842633 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.843457 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:38.843458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:38.843459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:38.843463 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:38.843474 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:38.843478 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:38.843521 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.843576 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.843595 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:41:38.843891 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.839334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33303 (local address 127.25.240.66:47123)
0924 13:41:38.839531 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.839562 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.840036 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.840264 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.840267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.840275 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:38.840754 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.840758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.841438 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.841439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.842316 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.842319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.842652 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.842653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.842757 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.843321 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:38.843322 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:38.843359 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:38.843387 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:38.843523 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:38.843526 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:38.843528 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.843573 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.843644 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260924 13:41:38.844370 13007 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33303
I20260924 13:41:38.844446 13007 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:38.844580 13007 heartbeater.cc:511] Master 127.25.240.126:33303 requested a full tablet report, sending...
I20260924 13:41:38.844904 12688 ts_manager.cc:194] Registered new tserver with Master: 7acbc99fdda947b5b27abc98932f2595 (127.25.240.66:32785)
I20260924 13:41:38.845274 12688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.66:47123
I20260924 13:41:38.846854 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 12879
I20260924 13:41:38.846941 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:41:38.847155 26561 external_mini_cluster.cc:1428] Setting key f3bd92e4a3a587e03d2eacde6236b925
I20260924 13:41:38.848031 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33303
--builtin_ntp_servers=127.25.240.84:41221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:38.926817 13010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:38.927009 13010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:38.927030 13010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:38.928088 13010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:38.928120 13010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:38.928144 13010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:41:38.929399 13010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:41221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:33303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:38.929625 13010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:38.929877 13010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:38.934710 13015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:38.934713 13018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:38.934724 13016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:38.934980 13010 server_base.cc:1034] running on GCE node
I20260924 13:41:38.935200 13010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:38.935441 13010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:38.936635 13010 hybrid_clock.cc:648] HybridClock initialized: now 1790257298936614 us; error 30 us; skew 500 ppm
I20260924 13:41:38.937897 13010 webserver.cc:460] Webserver started at http://127.25.240.67:44889/ using document root <none> and password file <none>
I20260924 13:41:38.938174 13010 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:38.938237 13010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:38.938382 13010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:38.939412 13010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "fcee9375610b40dab45abb9f3dea5549"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "36c98330fd2516b11994e6467bd8429e"
server_key_iv: "a604ac9fb4ff3737ac3a7f109797ee0c"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.939754 13010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "fcee9375610b40dab45abb9f3dea5549"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "36c98330fd2516b11994e6467bd8429e"
server_key_iv: "a604ac9fb4ff3737ac3a7f109797ee0c"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.941009 13010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:38.941697 13024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.941869 13010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:38.942001 13010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "fcee9375610b40dab45abb9f3dea5549"
format_stamp: "Formatted at 2026-09-24 13:41:38 on dist-test-slave-m5z1"
server_key: "36c98330fd2516b11994e6467bd8429e"
server_key_iv: "a604ac9fb4ff3737ac3a7f109797ee0c"
server_key_version: "encryptionkey@0"
I20260924 13:41:38.942083 13010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:38.956429 13010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:38.956722 13010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:38.956866 13010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:38.957078 13010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:38.957393 13010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:38.957445 13010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.957479 13010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:38.957506 13010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:38.963339 13010 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:35409
I20260924 13:41:38.963395 13137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:35409 every 8 connection(s)
I20260924 13:41:38.963728 13010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:41:38.968595 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.964447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:36611 (local address 127.25.240.126:33303)
0924 13:41:38.964526 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.964528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.964714 (+ 186us) server_negotiation.cc:407] Connection header received
0924 13:41:38.965185 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.965186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.965196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.965243 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:38.965966 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.966443 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.967367 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.967548 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.967597 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.968383 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:38.968384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:38.968384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:38.968388 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:38.968398 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:38.968401 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:38.968439 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.968519 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.968537 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 13:41:38.968828 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.964420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:33303 (local address 127.25.240.67:36611)
0924 13:41:38.964604 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.964635 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.965107 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.965302 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.965305 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.965312 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:38.965868 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.965873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.966519 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.966519 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.967298 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.967299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.967618 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.967619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.967722 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.968267 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:38.968267 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:38.968295 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:38.968317 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:38.968459 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:38.968461 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:38.968464 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.968514 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.968611 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260924 13:41:38.969297 13138 heartbeater.cc:347] Connected to a master server at 127.25.240.126:33303
I20260924 13:41:38.969388 13138 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:38.969573 13138 heartbeater.cc:511] Master 127.25.240.126:33303 requested a full tablet report, sending...
I20260924 13:41:38.969911 12688 ts_manager.cc:194] Registered new tserver with Master: fcee9375610b40dab45abb9f3dea5549 (127.25.240.67:35409)
I20260924 13:41:38.970242 12688 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.67:36611
I20260924 13:41:38.972383 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 13010
I20260924 13:41:38.972469 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:41:38.972626 26561 external_mini_cluster.cc:1428] Setting key 1ce3a91ad70f3c9b33becc6c51f268b4
I20260924 13:41:38.973398 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:41:38.977421 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.974866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.25.240.126:33303)
0924 13:41:38.974927 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.974929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.975049 (+ 120us) server_negotiation.cc:407] Connection header received
0924 13:41:38.975098 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.975099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.975107 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.975157 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:38.975277 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.975727 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.976454 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.976627 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.976674 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.977229 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:38.977230 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:38.977230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:38.977234 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:38.977248 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:38.977251 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:38.977319 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.977361 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.977370 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 13:41:38.978977 12688 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:41:38.980521 12688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:41:38.986526 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.982625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:32785 (local address 127.0.0.1:59072)
0924 13:41:38.982910 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.982944 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.983011 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.983205 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.983208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.983210 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:38.983277 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.983279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.984341 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.984342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.985320 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.985322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.986243 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.986244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.986315 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.986328 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.986403 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.986455 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":51,"threads_started":1}
I20260924 13:41:38.986538 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.982722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59072 (local address 127.25.240.66:32785)
0924 13:41:38.982919 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.982925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.982998 (+ 73us) server_negotiation.cc:407] Connection header received
0924 13:41:38.983091 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.983093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.983108 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.983200 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:38.983348 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.984266 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.985426 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.986169 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.986245 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.986336 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.986425 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.986460 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260924 13:41:38.986943 13150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.982555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:35409 (local address 127.0.0.1:46350)
0924 13:41:38.982902 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.982943 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.983007 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.983385 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.983386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.983388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:38.983437 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.983439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.984707 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.984708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.985770 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.985772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.986720 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.986721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.986780 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.986784 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.986829 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.986873 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":49,"threads_started":1}
I20260924 13:41:38.987218 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.982753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:36421 (local address 127.0.0.1:55480)
0924 13:41:38.983537 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:38.983546 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:38.983594 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:38.983765 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:38.983766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:38.983768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:38.983804 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.983805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.984967 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.984968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.986016 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:38.986019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.986906 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.986907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:38.986969 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.986974 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:41:38.987040 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:41:38.987077 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"thread_start_us":77,"threads_started":1}
I20260924 13:41:38.987252 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.982980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55480 (local address 127.25.240.65:36421)
0924 13:41:38.983141 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.983145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.983612 (+ 467us) server_negotiation.cc:407] Connection header received
0924 13:41:38.983660 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.983663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.983678 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.983760 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:38.983860 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.984890 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.986115 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.986825 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.986888 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.986964 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.987131 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.987172 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260924 13:41:38.987479 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:38.982623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46350 (local address 127.25.240.67:35409)
0924 13:41:38.983176 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:38.983180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:38.983190 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:41:38.983231 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:38.983233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:38.983248 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:38.983319 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:38.983500 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.984615 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.985847 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:38.986493 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:38.986559 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:38.987181 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:38.987203 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 13:41:38.987237 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":70,"threads_started":1}
I20260924 13:41:38.987350 12941 tablet_service.cc:1467] Processing CreateTablet for tablet 6883db4f95dd473f80387ef685ff13c8 (DEFAULT_TABLE table=test-table [id=1ab31b5ef4fb4f5192d7200f5a37a84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:38.987727 12941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6883db4f95dd473f80387ef685ff13c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:38.988080 13072 tablet_service.cc:1467] Processing CreateTablet for tablet 6883db4f95dd473f80387ef685ff13c8 (DEFAULT_TABLE table=test-table [id=1ab31b5ef4fb4f5192d7200f5a37a84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:38.988384 13072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6883db4f95dd473f80387ef685ff13c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:38.991120 13152 tablet_bootstrap.cc:492] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Bootstrap starting.
I20260924 13:41:38.991562 13153 tablet_bootstrap.cc:492] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: Bootstrap starting.
I20260924 13:41:38.991640 12810 tablet_service.cc:1467] Processing CreateTablet for tablet 6883db4f95dd473f80387ef685ff13c8 (DEFAULT_TABLE table=test-table [id=1ab31b5ef4fb4f5192d7200f5a37a84a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:38.991943 13152 tablet_bootstrap.cc:654] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:38.991961 12810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6883db4f95dd473f80387ef685ff13c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:38.992202 13153 tablet_bootstrap.cc:654] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:38.992274 13152 log.cc:824] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:38.992537 13153 log.cc:824] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:38.993007 13152 tablet_bootstrap.cc:492] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: No bootstrap required, opened a new log
I20260924 13:41:38.993084 13152 ts_tablet_manager.cc:1397] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:41:38.994788 13156 tablet_bootstrap.cc:492] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c: Bootstrap starting.
I20260924 13:41:38.995304 13153 tablet_bootstrap.cc:492] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: No bootstrap required, opened a new log
I20260924 13:41:38.995263 13152 raft_consensus.cc:348] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:38.995379 13153 ts_tablet_manager.cc:1397] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 13:41:38.995417 13152 raft_consensus.cc:374] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:38.995429 13156 tablet_bootstrap.cc:654] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:38.995460 13152 raft_consensus.cc:729] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acbc99fdda947b5b27abc98932f2595, State: Initialized, Role: FOLLOWER
I20260924 13:41:38.995590 13152 consensus_queue.cc:260] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:38.995723 13156 log.cc:824] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:38.995831 13152 ts_tablet_manager.cc:1428] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 13:41:38.995960 13007 heartbeater.cc:503] Master 127.25.240.126:33303 was elected leader, sending a full tablet report...
I20260924 13:41:38.996497 13156 tablet_bootstrap.cc:492] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c: No bootstrap required, opened a new log
I20260924 13:41:38.996608 13156 ts_tablet_manager.cc:1397] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:41:38.998061 13153 raft_consensus.cc:348] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:38.998221 13153 raft_consensus.cc:374] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:38.998262 13153 raft_consensus.cc:729] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcee9375610b40dab45abb9f3dea5549, State: Initialized, Role: FOLLOWER
I20260924 13:41:38.998373 13153 consensus_queue.cc:260] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:38.998643 13153 ts_tablet_manager.cc:1428] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 13:41:38.998673 13138 heartbeater.cc:503] Master 127.25.240.126:33303 was elected leader, sending a full tablet report...
I20260924 13:41:38.999191 13156 raft_consensus.cc:348] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:38.999361 13156 raft_consensus.cc:374] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:38.999399 13156 raft_consensus.cc:729] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304682c003de4b87b2f4fd26298d483c, State: Initialized, Role: FOLLOWER
I20260924 13:41:38.999496 13156 consensus_queue.cc:260] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:38.999720 13156 ts_tablet_manager.cc:1428] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:41:38.999781 12876 heartbeater.cc:503] Master 127.25.240.126:33303 was elected leader, sending a full tablet report...
I20260924 13:41:39.076836 13159 raft_consensus.cc:482] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:39.076954 13159 raft_consensus.cc:504] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:39.077392 13159 leader_election.cc:288] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 304682c003de4b87b2f4fd26298d483c (127.25.240.65:36421), 7acbc99fdda947b5b27abc98932f2595 (127.25.240.66:32785)
I20260924 13:41:39.080070 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:39.077543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:36421 (local address 127.25.240.67:58673)
0924 13:41:39.077617 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:39.077629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:39.077671 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:39.077808 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:39.077809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:39.077811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:39.077908 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:39.077910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.078535 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.078536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:39.079271 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:39.079273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.079802 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.079803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:39.079879 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:39.079882 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:39.079958 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:41:39.079998 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 13:41:39.080070 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:39.077613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:58673 (local address 127.25.240.65:36421)
0924 13:41:39.077674 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:39.077676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:39.077718 (+ 42us) server_negotiation.cc:407] Connection header received
0924 13:41:39.077749 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:39.077750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:39.077758 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:39.077797 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:39.077972 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.078452 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.079364 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.079722 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.079803 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:39.079812 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:39.079965 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 13:41:39.080005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:41:39.080458 12830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6883db4f95dd473f80387ef685ff13c8" candidate_uuid: "fcee9375610b40dab45abb9f3dea5549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304682c003de4b87b2f4fd26298d483c" is_pre_election: true
I20260924 13:41:39.080570 13161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:39.077841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:32785 (local address 127.25.240.67:46987)
0924 13:41:39.078024 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:39.078032 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:39.078100 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:39.078237 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:39.078238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:39.078239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:39.078313 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:39.078314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.078901 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.078901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:39.079625 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:39.079627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.080288 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.080289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:39.080367 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:39.080369 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:41:39.080449 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:41:39.080488 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":131,"threads_started":1}
I20260924 13:41:39.080559 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:39.077669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:46987 (local address 127.25.240.66:32785)
0924 13:41:39.077727 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:39.077729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:39.078088 (+ 359us) server_negotiation.cc:407] Connection header received
0924 13:41:39.078163 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:39.078164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:39.078173 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:39.078235 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:39.078376 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.078823 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.079710 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.080068 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.080147 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:39.080155 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:39.080458 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 13:41:39.080497 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 13:41:39.080626 12830 raft_consensus.cc:2454] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcee9375610b40dab45abb9f3dea5549 in term 0.
I20260924 13:41:39.080852 13026 leader_election.cc:302] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304682c003de4b87b2f4fd26298d483c, fcee9375610b40dab45abb9f3dea5549; no voters:
I20260924 13:41:39.080885 12961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6883db4f95dd473f80387ef685ff13c8" candidate_uuid: "fcee9375610b40dab45abb9f3dea5549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acbc99fdda947b5b27abc98932f2595" is_pre_election: true
I20260924 13:41:39.080978 13159 raft_consensus.cc:2781] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:41:39.081040 13159 raft_consensus.cc:482] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:41:39.081027 12961 raft_consensus.cc:2454] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcee9375610b40dab45abb9f3dea5549 in term 0.
I20260924 13:41:39.081064 13159 raft_consensus.cc:3037] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:39.081861 13159 raft_consensus.cc:504] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:39.081995 13159 leader_election.cc:288] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [CANDIDATE]: Term 1 election: Requested vote from peers 304682c003de4b87b2f4fd26298d483c (127.25.240.65:36421), 7acbc99fdda947b5b27abc98932f2595 (127.25.240.66:32785)
I20260924 13:41:39.082139 12830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6883db4f95dd473f80387ef685ff13c8" candidate_uuid: "fcee9375610b40dab45abb9f3dea5549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304682c003de4b87b2f4fd26298d483c"
I20260924 13:41:39.082151 12961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6883db4f95dd473f80387ef685ff13c8" candidate_uuid: "fcee9375610b40dab45abb9f3dea5549" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acbc99fdda947b5b27abc98932f2595"
I20260924 13:41:39.082227 12830 raft_consensus.cc:3037] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:39.082227 12961 raft_consensus.cc:3037] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:39.083173 12830 raft_consensus.cc:2454] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcee9375610b40dab45abb9f3dea5549 in term 1.
I20260924 13:41:39.083170 12961 raft_consensus.cc:2454] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcee9375610b40dab45abb9f3dea5549 in term 1.
I20260924 13:41:39.083380 13027 leader_election.cc:302] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7acbc99fdda947b5b27abc98932f2595, fcee9375610b40dab45abb9f3dea5549; no voters:
I20260924 13:41:39.083478 13159 raft_consensus.cc:2781] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:39.083621 13159 raft_consensus.cc:686] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 1 LEADER]: Becoming Leader. State: Replica: fcee9375610b40dab45abb9f3dea5549, State: Running, Role: LEADER
I20260924 13:41:39.083709 13159 consensus_queue.cc:237] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } }
I20260924 13:41:39.084497 12688 catalog_manager.cc:5495] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcee9375610b40dab45abb9f3dea5549 (127.25.240.67). New cstate: current_term: 1 leader_uuid: "fcee9375610b40dab45abb9f3dea5549" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcee9375610b40dab45abb9f3dea5549" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 35409 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 } health_report { overall_health: UNKNOWN } } }
W20260924 13:41:39.089248 13008 tablet.cc:2369] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:41:39.101224 13148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:39.098881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.25.240.67:35409)
0924 13:41:39.098947 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:39.098949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:39.098958 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:41:39.098997 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:39.098998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:39.099020 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:39.099099 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:41:39.099238 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.099708 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.100449 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:39.100645 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:39.100728 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:39.100880 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:41:39.100975 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:41:39.101080 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:39.101111 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 13:41:39.101119 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 13:41:39.102977 12961 raft_consensus.cc:1261] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 1 FOLLOWER]: Refusing update from remote peer fcee9375610b40dab45abb9f3dea5549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:39.103025 12830 raft_consensus.cc:1261] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 1 FOLLOWER]: Refusing update from remote peer fcee9375610b40dab45abb9f3dea5549: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:39.103323 13159 consensus_queue.cc:1035] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "304682c003de4b87b2f4fd26298d483c" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 36421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:39.103435 13162 consensus_queue.cc:1035] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acbc99fdda947b5b27abc98932f2595" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 32785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:39.106756 13167 mvcc.cc:205] Tried to move back new op lower bound from 7332893897123389440 to 7332893897047535616. Current Snapshot: MvccSnapshot[applied={T|T < 7332893897123389440}]
I20260924 13:41:39.107893 12688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:39.107990 12688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:39.108711 12688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32dc8896cadc4ceaa1a55ad343404510: Sending DeleteTablet for 3 replicas of tablet 6883db4f95dd473f80387ef685ff13c8
I20260924 13:41:39.109086 13072 tablet_service.cc:1514] Processing DeleteTablet for tablet 6883db4f95dd473f80387ef685ff13c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:39 UTC) from {username='slave'} at 127.0.0.1:46350
I20260924 13:41:39.109093 12941 tablet_service.cc:1514] Processing DeleteTablet for tablet 6883db4f95dd473f80387ef685ff13c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:39 UTC) from {username='slave'} at 127.0.0.1:59072
I20260924 13:41:39.109318 13176 tablet_replica.cc:330] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: stopping tablet replica
I20260924 13:41:39.109309 12810 tablet_service.cc:1514] Processing DeleteTablet for tablet 6883db4f95dd473f80387ef685ff13c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:39 UTC) from {username='slave'} at 127.0.0.1:55480
I20260924 13:41:39.109421 13176 raft_consensus.cc:2229] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:41:39.109505 13176 raft_consensus.cc:2258] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:39.109510 13177 tablet_replica.cc:330] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c: stopping tablet replica
I20260924 13:41:39.109715 13177 raft_consensus.cc:2229] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:39.109834 13177 raft_consensus.cc:2258] T 6883db4f95dd473f80387ef685ff13c8 P 304682c003de4b87b2f4fd26298d483c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:39.109980 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 12748
I20260924 13:41:39.110029 13176 ts_tablet_manager.cc:1905] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:39.111734 13175 tablet_replica.cc:330] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: stopping tablet replica
I20260924 13:41:39.111771 13176 ts_tablet_manager.cc:1918] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:39.111826 13176 log.cc:1196] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/6883db4f95dd473f80387ef685ff13c8
I20260924 13:41:39.111833 13175 raft_consensus.cc:2229] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:39.111928 13175 raft_consensus.cc:2258] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:39.112116 13176 ts_tablet_manager.cc:1939] T 6883db4f95dd473f80387ef685ff13c8 P fcee9375610b40dab45abb9f3dea5549: Deleting consensus metadata
I20260924 13:41:39.112360 13175 ts_tablet_manager.cc:1905] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:39.112522 12673 catalog_manager.cc:4841] TS fcee9375610b40dab45abb9f3dea5549 (127.25.240.67:35409): tablet 6883db4f95dd473f80387ef685ff13c8 (table test-table [id=1ab31b5ef4fb4f5192d7200f5a37a84a]) successfully deleted
I20260924 13:41:39.113925 13175 ts_tablet_manager.cc:1918] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:39.114001 13175 log.cc:1196] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/6883db4f95dd473f80387ef685ff13c8
I20260924 13:41:39.114248 13175 ts_tablet_manager.cc:1939] T 6883db4f95dd473f80387ef685ff13c8 P 7acbc99fdda947b5b27abc98932f2595: Deleting consensus metadata
I20260924 13:41:39.114634 12674 catalog_manager.cc:4841] TS 7acbc99fdda947b5b27abc98932f2595 (127.25.240.66:32785): tablet 6883db4f95dd473f80387ef685ff13c8 (table test-table [id=1ab31b5ef4fb4f5192d7200f5a37a84a]) successfully deleted
W20260924 13:41:39.115953 12673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:36421 (error 108)
I20260924 13:41:39.116153 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 12879
I20260924 13:41:39.116428 13147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:39.116302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:36421 (local address 127.0.0.1:55486)
0924 13:41:39.116357 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:39.116390 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 13:41:39.116528 12673 catalog_manager.cc:4584] TS 304682c003de4b87b2f4fd26298d483c (127.25.240.65:36421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6883db4f95dd473f80387ef685ff13c8: Network error: Client connection negotiation failed: client connection to 127.25.240.65:36421: connect: Connection refused (error 111)
I20260924 13:41:39.121147 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 13010
I20260924 13:41:39.125182 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 12657
2026-09-24T13:41:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (767 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:39 dist-test-slave-m5z1 krb5kdc[13181](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:39 dist-test-slave-m5z1 krb5kdc[13181](info): set up 2 sockets
Sep 24 13:41:39 dist-test-slave-m5z1 krb5kdc[13181](info): commencing operation
krb5kdc: starting...
W20260924 13:41:40.173673 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:41:40.292670 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:46087
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:46087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:40.372458 13194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:40.372658 13194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:40.372679 13194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:40.373879 13194 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:40.373924 13194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:40.373939 13194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:40.373950 13194 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:40.373962 13194 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:40.375566 13194 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:46087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:46087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:40.375805 13194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:40.376039 13194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:40.380895 13200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:40.380880 13202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:40.380924 13194 server_base.cc:1034] running on GCE node
W20260924 13:41:40.380884 13199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:40.381412 13194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:40.381786 13194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:40.382968 13194 hybrid_clock.cc:648] HybridClock initialized: now 1790257300382950 us; error 28 us; skew 500 ppm
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:40.385358 13194 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:41:40.385790 13194 webserver.cc:460] Webserver started at http://127.25.240.126:33575/ using document root <none> and password file <none>
I20260924 13:41:40.385975 13194 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:40.386016 13194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:40.386101 13194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:40.387132 13194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "8fdbebf5aa5e490db4d0759218555188"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "937149de6e59ea2a21dbe877578ba0d9"
server_key_iv: "462980563cb0f2598ab59c33ba5df0c6"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.387434 13194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "8fdbebf5aa5e490db4d0759218555188"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "937149de6e59ea2a21dbe877578ba0d9"
server_key_iv: "462980563cb0f2598ab59c33ba5df0c6"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.388703 13194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:40.389405 13209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.389640 13194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:40.389712 13194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "8fdbebf5aa5e490db4d0759218555188"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "937149de6e59ea2a21dbe877578ba0d9"
server_key_iv: "462980563cb0f2598ab59c33ba5df0c6"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.389766 13194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:40.399446 13194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:40.401325 13194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:40.401434 13194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:40.405225 13194 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:46087
I20260924 13:41:40.405290 13261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:46087 every 8 connection(s)
I20260924 13:41:40.405715 13194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:40.406657 13262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:40.407143 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 13194
I20260924 13:41:40.407258 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:40.407447 26561 external_mini_cluster.cc:1428] Setting key b95b63f44473c0000bf1c25d7da18af3
I20260924 13:41:40.409294 13262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188: Bootstrap starting.
I20260924 13:41:40.409941 13262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:40.410239 13262 log.cc:824] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:40.410955 13262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188: No bootstrap required, opened a new log
I20260924 13:41:40.412918 13262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdbebf5aa5e490db4d0759218555188" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46087 } }
I20260924 13:41:40.413075 13262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:40.413149 13262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fdbebf5aa5e490db4d0759218555188, State: Initialized, Role: FOLLOWER
I20260924 13:41:40.413308 13262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdbebf5aa5e490db4d0759218555188" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46087 } }
I20260924 13:41:40.413403 13262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:40.413446 13262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:40.413496 13262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:40.414063 13262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdbebf5aa5e490db4d0759218555188" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46087 } }
I20260924 13:41:40.414180 13262 leader_election.cc:302] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fdbebf5aa5e490db4d0759218555188; no voters:
I20260924 13:41:40.414361 13262 leader_election.cc:288] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:40.414419 13267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:40.414623 13262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:40.414620 13267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [term 1 LEADER]: Becoming Leader. State: Replica: 8fdbebf5aa5e490db4d0759218555188, State: Running, Role: LEADER
I20260924 13:41:40.414738 13267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdbebf5aa5e490db4d0759218555188" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46087 } }
I20260924 13:41:40.415160 13269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fdbebf5aa5e490db4d0759218555188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdbebf5aa5e490db4d0759218555188" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46087 } } }
I20260924 13:41:40.415247 13269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:40.415086 13268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fdbebf5aa5e490db4d0759218555188. Latest consensus state: current_term: 1 leader_uuid: "8fdbebf5aa5e490db4d0759218555188" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fdbebf5aa5e490db4d0759218555188" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46087 } } }
I20260924 13:41:40.415692 13276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:40.415733 13268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:40.416250 13276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:40.417920 13276 catalog_manager.cc:1347] Generated new cluster ID: b3578c9b80274488830135bc927b42e8
I20260924 13:41:40.417972 13276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:40.418567 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.407990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50706 (local address 127.25.240.126:46087)
0924 13:41:40.408179 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.408188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.408221 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:41:40.408635 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.408642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.408686 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.408854 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:40.409586 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.410582 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.411411 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.411820 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.411936 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.414527 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:40.414531 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:40.414531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:40.414543 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:40.416696 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.417138 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.417141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.417142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.417246 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.417497 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.417499 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.417499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.417833 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:40.417986 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.418251 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.418385 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260924 13:41:40.425313 13276 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:40.425896 13276 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:40.428927 13276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188: Generated new TSK 0
I20260924 13:41:40.429082 13276 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.240.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
Entry for principal kudu/127.25.240.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab.
I20260924 13:41:40.460510 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46087
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:40.542685 13288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:40.542915 13288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:40.542954 13288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:40.544139 13288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:40.544195 13288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:40.544240 13288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:41:40.545784 13288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.65.keytab
--principal=kudu/127.25.240.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:40.546037 13288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:40.546254 13288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:40.551101 13296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:40.551203 13293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:40.551113 13294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:40.551146 13288 server_base.cc:1034] running on GCE node
I20260924 13:41:40.551532 13288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:40.551756 13288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:40.552897 13288 hybrid_clock.cc:648] HybridClock initialized: now 1790257300552887 us; error 27 us; skew 500 ppm
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:40.555348 13288 init.cc:377] Logged in from keytab as kudu/127.25.240.65@KRBTEST.COM (short username kudu)
I20260924 13:41:40.555757 13288 webserver.cc:460] Webserver started at http://127.25.240.65:44737/ using document root <none> and password file <none>
I20260924 13:41:40.555989 13288 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:40.556033 13288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:40.556185 13288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:40.557178 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "2615152c4d494ac8ac723c6b978ec2ad"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "0038f5bda9375635b0b4877f1a4462d1"
server_key_iv: "76bc85e1515c431fda85683c890a71c1"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.557523 13288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "2615152c4d494ac8ac723c6b978ec2ad"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "0038f5bda9375635b0b4877f1a4462d1"
server_key_iv: "76bc85e1515c431fda85683c890a71c1"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.558847 13288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:41:40.559544 13303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.559713 13288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:41:40.559810 13288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "2615152c4d494ac8ac723c6b978ec2ad"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "0038f5bda9375635b0b4877f1a4462d1"
server_key_iv: "76bc85e1515c431fda85683c890a71c1"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.559878 13288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:40.569828 13288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:40.571600 13288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:40.571746 13288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:40.571955 13288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:40.572234 13288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:40.572283 13288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.572322 13288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:40.572360 13288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.578311 13288 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:39313
I20260924 13:41:40.578352 13416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:39313 every 8 connection(s)
I20260924 13:41:40.578639 13288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.65@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:40.585496 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 13288
I20260924 13:41:40.585639 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:41:40.585863 26561 external_mini_cluster.cc:1428] Setting key 2a12df97831d7c1f9a9ead55306e48fb
I20260924 13:41:40.587996 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.579320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:53929 (local address 127.25.240.126:46087)
0924 13:41:40.579382 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.579384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.579581 (+ 197us) server_negotiation.cc:407] Connection header received
0924 13:41:40.580043 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.580044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.580052 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.580104 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:40.580969 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.581461 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.582550 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.582787 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.582871 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.584925 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:40.584927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:40.584927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:40.584934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:40.586589 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.587012 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.587013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.587013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.587069 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.587320 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.587321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.587321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.587438 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:40.587514 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.587771 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.587904 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 13:41:40.588245 13419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.579295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46087 (local address 127.25.240.65:53929)
0924 13:41:40.579462 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:40.579502 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:40.579965 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:40.580158 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:40.580161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:40.580704 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:40.580898 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.580902 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.581548 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.581549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.582453 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.582455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.582914 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.582915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.583020 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.583467 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:40.583475 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:40.584838 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:40.586678 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:40.586685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:40.586687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:40.586922 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:40.587147 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:40.587147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:40.587148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:40.587250 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:40.587505 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:40.587511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:40.587646 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 13:41:40.587750 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 13:41:40.587904 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260924 13:41:40.588755 13417 heartbeater.cc:347] Connected to a master server at 127.25.240.126:46087
I20260924 13:41:40.588852 13417 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:40.589030 13417 heartbeater.cc:511] Master 127.25.240.126:46087 requested a full tablet report, sending...
I20260924 13:41:40.589592 13226 ts_manager.cc:194] Registered new tserver with Master: 2615152c4d494ac8ac723c6b978ec2ad (127.25.240.65:39313)
I20260924 13:41:40.590221 13226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.65@KRBTEST.COM'} at 127.25.240.65:53929
WARNING: no policy specified for kudu/127.25.240.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
Entry for principal kudu/127.25.240.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab.
I20260924 13:41:40.611824 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46087
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:40.694597 13422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:40.694789 13422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:40.694813 13422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:40.696089 13422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:40.696130 13422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:40.696174 13422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:41:40.697674 13422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.66.keytab
--principal=kudu/127.25.240.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:40.697898 13422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:40.698120 13422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:40.703126 13427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:40.703329 13422 server_base.cc:1034] running on GCE node
W20260924 13:41:40.703153 13430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:40.703135 13428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:40.703646 13422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:40.703852 13422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:40.705009 13422 hybrid_clock.cc:648] HybridClock initialized: now 1790257300705001 us; error 40 us; skew 500 ppm
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:40.707511 13422 init.cc:377] Logged in from keytab as kudu/127.25.240.66@KRBTEST.COM (short username kudu)
I20260924 13:41:40.707993 13422 webserver.cc:460] Webserver started at http://127.25.240.66:38097/ using document root <none> and password file <none>
I20260924 13:41:40.708221 13422 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:40.708284 13422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:40.708424 13422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:40.709425 13422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "e581e51ea23841a89566ac543474c34a"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "bb2b5d83a1eb384a44848e3dab6d6864"
server_key_iv: "6b8280ea314fbde8a780d5c39b5e2773"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.709800 13422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "e581e51ea23841a89566ac543474c34a"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "bb2b5d83a1eb384a44848e3dab6d6864"
server_key_iv: "6b8280ea314fbde8a780d5c39b5e2773"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.711112 13422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:40.711882 13437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.712052 13422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:40.712144 13422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "e581e51ea23841a89566ac543474c34a"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "bb2b5d83a1eb384a44848e3dab6d6864"
server_key_iv: "6b8280ea314fbde8a780d5c39b5e2773"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.712219 13422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:40.722389 13422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:40.724251 13422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:40.724406 13422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:40.724614 13422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:40.724907 13422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:40.724956 13422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.724990 13422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:40.725018 13422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.731087 13422 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:46293
I20260924 13:41:40.731122 13550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:46293 every 8 connection(s)
I20260924 13:41:40.731436 13422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:41:40.736402 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 13422
I20260924 13:41:40.736519 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:41:40.736753 26561 external_mini_cluster.cc:1428] Setting key 910177a98bc112606eaea4178147424e
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.66@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:40.741652 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.732201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:52399 (local address 127.25.240.126:46087)
0924 13:41:40.732267 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.732268 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.732477 (+ 209us) server_negotiation.cc:407] Connection header received
0924 13:41:40.732999 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.733001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.733013 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.733068 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:40.734065 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.734557 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.735542 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.735732 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.735814 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.738414 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:40.738416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:40.738417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:40.738425 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:40.740059 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.740592 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.740593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.740593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.740649 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.740950 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.740950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.740951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.741053 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:40.741102 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.741465 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.741537 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:41:40.741959 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.732188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46087 (local address 127.25.240.66:52399)
0924 13:41:40.732363 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:40.732397 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:40.732894 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:40.733149 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:40.733153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:40.733780 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:40.733966 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.733971 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.734649 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.734650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.735447 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.735449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.735822 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.735824 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.735945 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.736487 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:40.736499 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:40.738285 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:40.740146 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:40.740153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:40.740155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:40.740512 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:40.740722 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:40.740723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:40.740724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:40.740879 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:40.741114 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:40.741121 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:40.741303 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 13:41:40.741437 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 13:41:40.741656 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260924 13:41:40.742550 13551 heartbeater.cc:347] Connected to a master server at 127.25.240.126:46087
I20260924 13:41:40.742658 13551 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:40.742888 13551 heartbeater.cc:511] Master 127.25.240.126:46087 requested a full tablet report, sending...
I20260924 13:41:40.743266 13226 ts_manager.cc:194] Registered new tserver with Master: e581e51ea23841a89566ac543474c34a (127.25.240.66:46293)
I20260924 13:41:40.743631 13226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.66@KRBTEST.COM'} at 127.25.240.66:52399
WARNING: no policy specified for kudu/127.25.240.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
Entry for principal kudu/127.25.240.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab.
I20260924 13:41:40.764583 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46087
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:40.844135 13556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:40.844340 13556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:40.844364 13556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:40.845434 13556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:40.845465 13556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:40.845491 13556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:41:40.846846 13556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.67.keytab
--principal=kudu/127.25.240.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:46087
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:40.847023 13556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:40.847198 13556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:40.852160 13564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:40.852152 13562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:40.852171 13561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:40.852319 13556 server_base.cc:1034] running on GCE node
I20260924 13:41:40.852629 13556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:40.852862 13556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:40.854024 13556 hybrid_clock.cc:648] HybridClock initialized: now 1790257300854013 us; error 27 us; skew 500 ppm
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:40.856665 13556 init.cc:377] Logged in from keytab as kudu/127.25.240.67@KRBTEST.COM (short username kudu)
I20260924 13:41:40.857215 13556 webserver.cc:460] Webserver started at http://127.25.240.67:39313/ using document root <none> and password file <none>
I20260924 13:41:40.857478 13556 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:40.857578 13556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:40.857728 13556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:40.858794 13556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "2d1bba23bead478d9ae50d02adf24299"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "3a67b3d9d4cc3839eca12d0d57cf6daa"
server_key_iv: "55f60e0acfa3ef6257722e36094166fc"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.859091 13556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "2d1bba23bead478d9ae50d02adf24299"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "3a67b3d9d4cc3839eca12d0d57cf6daa"
server_key_iv: "55f60e0acfa3ef6257722e36094166fc"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.860497 13556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:40.861441 13571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.861660 13556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:41:40.861753 13556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "2d1bba23bead478d9ae50d02adf24299"
format_stamp: "Formatted at 2026-09-24 13:41:40 on dist-test-slave-m5z1"
server_key: "3a67b3d9d4cc3839eca12d0d57cf6daa"
server_key_iv: "55f60e0acfa3ef6257722e36094166fc"
server_key_version: "encryptionkey@0"
I20260924 13:41:40.861826 13556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:40.875672 13556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:40.877475 13556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:40.877660 13556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:40.877843 13556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:40.878132 13556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:40.878166 13556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.878186 13556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:40.878199 13556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:40.883723 13556 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:34457
I20260924 13:41:40.883793 13684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:34457 every 8 connection(s)
I20260924 13:41:40.884106 13556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:41:40.889079 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 13556
I20260924 13:41:40.889200 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:41:40.889433 26561 external_mini_cluster.cc:1428] Setting key 104d99f3fee61213c68b07277de54780
Sep 24 13:41:40 dist-test-slave-m5z1 krb5kdc[13181](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257300, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.67@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:40.893548 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.884830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:56531 (local address 127.25.240.126:46087)
0924 13:41:40.884960 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.884963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.885106 (+ 143us) server_negotiation.cc:407] Connection header received
0924 13:41:40.885644 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.885645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.885657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.885720 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:40.886623 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.887120 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.888080 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.888292 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.888363 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.890787 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:40.890788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:40.890788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:40.890794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:40.892133 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.892496 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.892496 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.892497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.892546 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.893011 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.893012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.893012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.893129 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:40.893205 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.893427 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.893486 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260924 13:41:40.893857 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.884819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:46087 (local address 127.25.240.67:56531)
0924 13:41:40.884991 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:40.885031 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:40.885509 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:40.885791 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:40.885794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:40.886336 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:40.886522 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.886526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.887208 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.887209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.887994 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.887996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.888369 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.888370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.888486 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.888956 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:41:40.888968 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:40.890717 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:40.892194 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:40.892200 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:40.892202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:40.892434 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:40.892669 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:41:40.892670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:41:40.892670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:41:40.892787 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:41:40.893195 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:40.893199 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:40.893312 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 13:41:40.893400 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 13:41:40.893527 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260924 13:41:40.894299 13685 heartbeater.cc:347] Connected to a master server at 127.25.240.126:46087
I20260924 13:41:40.894397 13685 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:40.894563 13685 heartbeater.cc:511] Master 127.25.240.126:46087 requested a full tablet report, sending...
I20260924 13:41:40.895009 13226 ts_manager.cc:194] Registered new tserver with Master: 2d1bba23bead478d9ae50d02adf24299 (127.25.240.67:34457)
I20260924 13:41:40.895346 13226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.240.67@KRBTEST.COM'} at 127.25.240.67:56531
I20260924 13:41:40.901347 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:41:40.908438 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.902754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.25.240.126:46087)
0924 13:41:40.902812 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.902814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.902947 (+ 133us) server_negotiation.cc:407] Connection header received
0924 13:41:40.902972 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.902972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.902983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.903046 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:40.903557 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.904092 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.904789 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.905020 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.905081 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.906197 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:40.906198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:40.906198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:40.906205 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:40.907521 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.907813 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.907814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.907815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.907872 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:40.908035 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:40.908036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:40.908036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:40.908158 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:40.908249 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.908319 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.908379 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 13:41:40.909811 13226 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50716:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:41:40.911387 13226 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:41:40.917896 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.913501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:39313 (local address 127.0.0.1:36280)
0924 13:41:40.913693 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:40.913751 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:40.913826 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:40.914110 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:40.914113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:40.914116 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:40.914185 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.914187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.915818 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.915819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.916900 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.916903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.917594 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.917595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.917674 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.917687 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:41:40.917761 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 13:41:40.917811 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":54,"thread_start_us":86,"threads_started":1}
I20260924 13:41:40.917896 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.913714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.25.240.65:39313)
0924 13:41:40.913912 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.913917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.913929 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:41:40.913991 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.913993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.914011 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.914097 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:40.914283 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.915692 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.916982 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.917477 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.917600 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.917662 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.917775 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.917813 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
I20260924 13:41:40.918058 13694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.913501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46293 (local address 127.0.0.1:42920)
0924 13:41:40.913943 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:40.913952 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:40.913995 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:40.914392 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:40.914392 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:40.914394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:40.914424 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.914425 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.915519 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.915520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.916892 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.916894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.917810 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.917811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.917869 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.917873 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:40.917901 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 13:41:40.917948 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":49,"spinlock_wait_cycles":768,"thread_start_us":94,"threads_started":1}
I20260924 13:41:40.918380 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.913780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42920 (local address 127.25.240.66:46293)
0924 13:41:40.914204 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.914208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.914235 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:41:40.914271 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.914273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.914289 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.914395 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:40.914491 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.915421 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.916977 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.917713 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.918159 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.918224 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.918250 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.918296 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":75,"threads_started":1}
I20260924 13:41:40.918633 13351 tablet_service.cc:1467] Processing CreateTablet for tablet db84722a5c7f42869f484ff00bf716b7 (DEFAULT_TABLE table=test-table [id=8178fb192de84608bb474681cc30ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:40.919035 13351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db84722a5c7f42869f484ff00bf716b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:40.919041 13485 tablet_service.cc:1467] Processing CreateTablet for tablet db84722a5c7f42869f484ff00bf716b7 (DEFAULT_TABLE table=test-table [id=8178fb192de84608bb474681cc30ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:40.919356 13485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db84722a5c7f42869f484ff00bf716b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:40.919821 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.913522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:34457 (local address 127.0.0.1:43970)
0924 13:41:40.913945 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:40.913957 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:40.914012 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:40.914254 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:40.914254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:40.914256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:40.914311 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.914313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.915833 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.915835 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.918589 (+ 2754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.918592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.919581 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.919582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.919639 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.919643 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:40.919702 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 13:41:40.919745 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":66,"thread_start_us":75,"threads_started":1}
I20260924 13:41:40.919852 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.913793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.25.240.67:34457)
0924 13:41:40.914097 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.914101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.914110 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:41:40.914143 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.914145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.914160 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.914243 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:40.914361 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.915697 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.918685 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.919409 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.919492 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.919544 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.919730 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.919772 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":83,"threads_started":1}
I20260924 13:41:40.920683 13619 tablet_service.cc:1467] Processing CreateTablet for tablet db84722a5c7f42869f484ff00bf716b7 (DEFAULT_TABLE table=test-table [id=8178fb192de84608bb474681cc30ce76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:40.921015 13619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db84722a5c7f42869f484ff00bf716b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:40.922189 13699 tablet_bootstrap.cc:492] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad: Bootstrap starting.
I20260924 13:41:40.922717 13700 tablet_bootstrap.cc:492] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Bootstrap starting.
I20260924 13:41:40.922925 13699 tablet_bootstrap.cc:654] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:40.923418 13700 tablet_bootstrap.cc:654] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:40.923522 13699 log.cc:824] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:40.923940 13700 log.cc:824] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:40.924265 13702 tablet_bootstrap.cc:492] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: Bootstrap starting.
I20260924 13:41:40.924358 13699 tablet_bootstrap.cc:492] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad: No bootstrap required, opened a new log
I20260924 13:41:40.924444 13699 ts_tablet_manager.cc:1397] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:41:40.924629 13700 tablet_bootstrap.cc:492] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: No bootstrap required, opened a new log
I20260924 13:41:40.924695 13700 ts_tablet_manager.cc:1397] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:41:40.924916 13702 tablet_bootstrap.cc:654] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:40.925441 13702 log.cc:824] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:40.926079 13702 tablet_bootstrap.cc:492] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: No bootstrap required, opened a new log
I20260924 13:41:40.926156 13702 ts_tablet_manager.cc:1397] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:41:40.926687 13699 raft_consensus.cc:348] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.926687 13700 raft_consensus.cc:348] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.926869 13699 raft_consensus.cc:374] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:40.926869 13700 raft_consensus.cc:374] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:40.926910 13699 raft_consensus.cc:729] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2615152c4d494ac8ac723c6b978ec2ad, State: Initialized, Role: FOLLOWER
I20260924 13:41:40.926918 13700 raft_consensus.cc:729] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e581e51ea23841a89566ac543474c34a, State: Initialized, Role: FOLLOWER
I20260924 13:41:40.927032 13700 consensus_queue.cc:260] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.927058 13699 consensus_queue.cc:260] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.927305 13699 ts_tablet_manager.cc:1428] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:41:40.927305 13700 ts_tablet_manager.cc:1428] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:41:40.927459 13417 heartbeater.cc:503] Master 127.25.240.126:46087 was elected leader, sending a full tablet report...
I20260924 13:41:40.927470 13551 heartbeater.cc:503] Master 127.25.240.126:46087 was elected leader, sending a full tablet report...
I20260924 13:41:40.927970 13702 raft_consensus.cc:348] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.928138 13702 raft_consensus.cc:374] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:40.928174 13702 raft_consensus.cc:729] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d1bba23bead478d9ae50d02adf24299, State: Initialized, Role: FOLLOWER
I20260924 13:41:40.928282 13702 consensus_queue.cc:260] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.928562 13702 ts_tablet_manager.cc:1428] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 13:41:40.928583 13685 heartbeater.cc:503] Master 127.25.240.126:46087 was elected leader, sending a full tablet report...
W20260924 13:41:40.982019 13552 tablet.cc:2369] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:41:40.986857 13707 raft_consensus.cc:482] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:40.986991 13707 raft_consensus.cc:504] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.987301 13707 leader_election.cc:288] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2615152c4d494ac8ac723c6b978ec2ad (127.25.240.65:39313), e581e51ea23841a89566ac543474c34a (127.25.240.66:46293)
I20260924 13:41:40.991382 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.987575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:37191 (local address 127.25.240.66:46293)
0924 13:41:40.987624 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.987626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.987872 (+ 246us) server_negotiation.cc:407] Connection header received
0924 13:41:40.987952 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.987953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.987962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.988020 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:40.988148 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.988833 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.990329 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.990925 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.991009 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.991018 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.991254 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.991304 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:41:40.991375 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.987605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:60371 (local address 127.25.240.65:39313)
0924 13:41:40.987649 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:40.987650 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:40.987659 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:41:40.987689 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:40.987690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:40.987697 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:40.987750 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:40.987954 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.988722 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.990329 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.990935 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.991009 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.991018 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:40.991256 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 13:41:40.991304 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:41:40.991395 13687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.987482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:39313 (local address 127.25.240.67:60371)
0924 13:41:40.987547 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:40.987559 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:40.987603 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:40.987772 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:40.987773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:40.987775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:40.987857 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.987859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.988817 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.988818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.990218 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.990221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.991039 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.991040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.991151 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.991153 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:41:40.991219 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:41:40.991282 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":124}
I20260924 13:41:40.991407 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:40.987482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:46293 (local address 127.25.240.67:37191)
0924 13:41:40.987774 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:40.987784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:40.987884 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:40.988029 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:40.988030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:40.988031 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:40.988083 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.988084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.988913 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.988914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.990218 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:40.990221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:40.991039 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:40.991040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:40.991148 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:40.991153 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 13:41:40.991219 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 13:41:40.991282 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":93,"thread_start_us":114,"threads_started":1}
I20260924 13:41:40.991798 13371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db84722a5c7f42869f484ff00bf716b7" candidate_uuid: "2d1bba23bead478d9ae50d02adf24299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2615152c4d494ac8ac723c6b978ec2ad" is_pre_election: true
I20260924 13:41:40.991987 13371 raft_consensus.cc:2454] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d1bba23bead478d9ae50d02adf24299 in term 0.
I20260924 13:41:40.992107 13505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db84722a5c7f42869f484ff00bf716b7" candidate_uuid: "2d1bba23bead478d9ae50d02adf24299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e581e51ea23841a89566ac543474c34a" is_pre_election: true
I20260924 13:41:40.992277 13505 raft_consensus.cc:2454] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d1bba23bead478d9ae50d02adf24299 in term 0.
I20260924 13:41:40.992264 13572 leader_election.cc:302] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2615152c4d494ac8ac723c6b978ec2ad, 2d1bba23bead478d9ae50d02adf24299; no voters:
I20260924 13:41:40.992406 13707 raft_consensus.cc:2781] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:41:40.992457 13707 raft_consensus.cc:482] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:41:40.992493 13707 raft_consensus.cc:3037] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:40.993286 13707 raft_consensus.cc:504] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.993444 13707 leader_election.cc:288] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [CANDIDATE]: Term 1 election: Requested vote from peers 2615152c4d494ac8ac723c6b978ec2ad (127.25.240.65:39313), e581e51ea23841a89566ac543474c34a (127.25.240.66:46293)
I20260924 13:41:40.993614 13505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db84722a5c7f42869f484ff00bf716b7" candidate_uuid: "2d1bba23bead478d9ae50d02adf24299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e581e51ea23841a89566ac543474c34a"
I20260924 13:41:40.993649 13371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db84722a5c7f42869f484ff00bf716b7" candidate_uuid: "2d1bba23bead478d9ae50d02adf24299" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2615152c4d494ac8ac723c6b978ec2ad"
I20260924 13:41:40.993696 13505 raft_consensus.cc:3037] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:40.993712 13371 raft_consensus.cc:3037] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:40.994635 13505 raft_consensus.cc:2454] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d1bba23bead478d9ae50d02adf24299 in term 1.
I20260924 13:41:40.994628 13371 raft_consensus.cc:2454] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d1bba23bead478d9ae50d02adf24299 in term 1.
I20260924 13:41:40.994853 13575 leader_election.cc:302] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d1bba23bead478d9ae50d02adf24299, e581e51ea23841a89566ac543474c34a; no voters:
I20260924 13:41:40.994989 13707 raft_consensus.cc:2781] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:40.995169 13707 raft_consensus.cc:686] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 1 LEADER]: Becoming Leader. State: Replica: 2d1bba23bead478d9ae50d02adf24299, State: Running, Role: LEADER
I20260924 13:41:40.995263 13707 consensus_queue.cc:237] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } }
I20260924 13:41:40.996106 13226 catalog_manager.cc:5495] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d1bba23bead478d9ae50d02adf24299 (127.25.240.67). New cstate: current_term: 1 leader_uuid: "2d1bba23bead478d9ae50d02adf24299" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d1bba23bead478d9ae50d02adf24299" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 34457 } health_report { overall_health: HEALTHY } } }
I20260924 13:41:41.008733 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.006655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.25.240.67:34457)
0924 13:41:41.006738 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.006740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.006757 (+ 17us) server_negotiation.cc:407] Connection header received
0924 13:41:41.006785 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.006786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.006803 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.006869 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:41:41.006983 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.007426 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.008122 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.008297 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.008344 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.008477 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:41:41.008573 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:41:41.008631 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.008664 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.008678 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 13:41:41.010398 13505 raft_consensus.cc:1261] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 1 FOLLOWER]: Refusing update from remote peer 2d1bba23bead478d9ae50d02adf24299: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:41.010414 13371 raft_consensus.cc:1261] T db84722a5c7f42869f484ff00bf716b7 P 2615152c4d494ac8ac723c6b978ec2ad [term 1 FOLLOWER]: Refusing update from remote peer 2d1bba23bead478d9ae50d02adf24299: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:41.010756 13707 consensus_queue.cc:1035] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e581e51ea23841a89566ac543474c34a" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:41.010929 13709 consensus_queue.cc:1035] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2615152c4d494ac8ac723c6b978ec2ad" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:41.012979 13712 mvcc.cc:205] Tried to move back new op lower bound from 7332893904936210432 to 7332893904877359104. Current Snapshot: MvccSnapshot[applied={T|T < 7332893904936210432}]
I20260924 13:41:41.013262 13713 mvcc.cc:205] Tried to move back new op lower bound from 7332893904936210432 to 7332893904877359104. Current Snapshot: MvccSnapshot[applied={T|T < 7332893904936210432}]
I20260924 13:41:41.013684 13714 mvcc.cc:205] Tried to move back new op lower bound from 7332893904936210432 to 7332893904877359104. Current Snapshot: MvccSnapshot[applied={T|T < 7332893904936210432}]
I20260924 13:41:41.015466 13226 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:41.015555 13226 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50716:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:41.016289 13226 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8fdbebf5aa5e490db4d0759218555188: Sending DeleteTablet for 3 replicas of tablet db84722a5c7f42869f484ff00bf716b7
I20260924 13:41:41.016654 13351 tablet_service.cc:1514] Processing DeleteTablet for tablet db84722a5c7f42869f484ff00bf716b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:41 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:36280
I20260924 13:41:41.016657 13485 tablet_service.cc:1514] Processing DeleteTablet for tablet db84722a5c7f42869f484ff00bf716b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:41 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:42920
I20260924 13:41:41.016852 13619 tablet_service.cc:1514] Processing DeleteTablet for tablet db84722a5c7f42869f484ff00bf716b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:41 UTC) from {username='kudu', principal='kudu/127.25.240.126@KRBTEST.COM'} at 127.0.0.1:43970
I20260924 13:41:41.017113 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 13288
I20260924 13:41:41.017055 13722 tablet_replica.cc:330] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: stopping tablet replica
I20260924 13:41:41.018064 13722 raft_consensus.cc:2229] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:41.018175 13722 raft_consensus.cc:2258] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:41.018579 13722 ts_tablet_manager.cc:1905] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:41.020548 13722 ts_tablet_manager.cc:1918] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:41.020624 13722 log.cc:1196] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/db84722a5c7f42869f484ff00bf716b7
I20260924 13:41:41.020875 13722 ts_tablet_manager.cc:1939] T db84722a5c7f42869f484ff00bf716b7 P e581e51ea23841a89566ac543474c34a: Deleting consensus metadata
I20260924 13:41:41.021299 13213 catalog_manager.cc:4841] TS e581e51ea23841a89566ac543474c34a (127.25.240.66:46293): tablet db84722a5c7f42869f484ff00bf716b7 (table test-table [id=8178fb192de84608bb474681cc30ce76]) successfully deleted
I20260924 13:41:41.021673 13724 tablet_replica.cc:330] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: stopping tablet replica
I20260924 13:41:41.021777 13724 raft_consensus.cc:2229] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:41:41.021898 13724 raft_consensus.cc:2258] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:41.022258 13724 ts_tablet_manager.cc:1905] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 13:41:41.023874 13210 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:39313 (error 108)
I20260924 13:41:41.024111 13724 ts_tablet_manager.cc:1918] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:41.024178 13724 log.cc:1196] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/db84722a5c7f42869f484ff00bf716b7
I20260924 13:41:41.024348 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.024210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:39313 (local address 127.0.0.1:36288)
0924 13:41:41.024283 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.024316 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:39313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 13:41:41.024434 13724 ts_tablet_manager.cc:1939] T db84722a5c7f42869f484ff00bf716b7 P 2d1bba23bead478d9ae50d02adf24299: Deleting consensus metadata
I20260924 13:41:41.024426 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 13422
W20260924 13:41:41.025005 13210 catalog_manager.cc:4584] TS 2615152c4d494ac8ac723c6b978ec2ad (127.25.240.65:39313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet db84722a5c7f42869f484ff00bf716b7: Network error: Client connection negotiation failed: client connection to 127.25.240.65:39313: connect: Connection refused (error 111)
I20260924 13:41:41.025425 13212 catalog_manager.cc:4841] TS 2d1bba23bead478d9ae50d02adf24299 (127.25.240.67:34457): tablet db84722a5c7f42869f484ff00bf716b7 (table test-table [id=8178fb192de84608bb474681cc30ce76]) successfully deleted
I20260924 13:41:41.029784 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 13556
I20260924 13:41:41.034443 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 13194
2026-09-24T13:41:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1909 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T13:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:41Z Disabled control of system clock
I20260924 13:41:41.061106 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39771
--webserver_interface=127.25.240.126
--webserver_port=0
--builtin_ntp_servers=127.25.240.84:37897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:41.139914 13729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:41.140120 13729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:41.140141 13729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:41.141193 13729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:41.141224 13729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:41.141237 13729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:41.141249 13729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:41.141261 13729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:41.142518 13729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:39771
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:41.142719 13729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:41.142938 13729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:41:41.147887 13729 server_base.cc:1034] running on GCE node
W20260924 13:41:41.147871 13737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:41.147858 13735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:41.147873 13734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:41.148368 13729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:41.148666 13729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:41.149806 13729 hybrid_clock.cc:648] HybridClock initialized: now 1790257301149792 us; error 29 us; skew 500 ppm
I20260924 13:41:41.151065 13729 webserver.cc:460] Webserver started at http://127.25.240.126:34985/ using document root <none> and password file <none>
I20260924 13:41:41.151302 13729 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:41.151350 13729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:41.151515 13729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:41.152570 13729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "15def8b08c784724b55300d4c13e5e0d"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "0ba2139c9f2d44abffd8afa75c71638f"
server_key_iv: "b0ac4fac4b5bb9e4d9e9fd4b583f8e39"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.152930 13729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "15def8b08c784724b55300d4c13e5e0d"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "0ba2139c9f2d44abffd8afa75c71638f"
server_key_iv: "b0ac4fac4b5bb9e4d9e9fd4b583f8e39"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.154277 13729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:41.154990 13743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.155146 13729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:41.155238 13729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "15def8b08c784724b55300d4c13e5e0d"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "0ba2139c9f2d44abffd8afa75c71638f"
server_key_iv: "b0ac4fac4b5bb9e4d9e9fd4b583f8e39"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.155314 13729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:41.174001 13729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:41.174335 13729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:41.174481 13729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:41.178480 13729 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:39771
I20260924 13:41:41.178517 13795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:39771 every 8 connection(s)
I20260924 13:41:41.178864 13729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:41.179838 13796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:41.182101 13796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d: Bootstrap starting.
I20260924 13:41:41.182667 13796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:41.182958 13796 log.cc:824] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:41.183604 13796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d: No bootstrap required, opened a new log
I20260924 13:41:41.185168 13796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15def8b08c784724b55300d4c13e5e0d" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39771 } }
I20260924 13:41:41.185324 13796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:41.185371 13796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15def8b08c784724b55300d4c13e5e0d, State: Initialized, Role: FOLLOWER
I20260924 13:41:41.185441 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 13729
I20260924 13:41:41.185587 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:41.185626 13796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15def8b08c784724b55300d4c13e5e0d" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39771 } }
I20260924 13:41:41.185729 13796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:41.185772 13796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:41.185789 26561 external_mini_cluster.cc:1428] Setting key 218839b6b5076e81d5f2858d765b49a5
I20260924 13:41:41.185827 13796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:41.186539 13796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15def8b08c784724b55300d4c13e5e0d" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39771 } }
I20260924 13:41:41.186651 13796 leader_election.cc:302] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15def8b08c784724b55300d4c13e5e0d; no voters:
I20260924 13:41:41.186795 13796 leader_election.cc:288] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:41.186823 13801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:41.186987 13801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [term 1 LEADER]: Becoming Leader. State: Replica: 15def8b08c784724b55300d4c13e5e0d, State: Running, Role: LEADER
I20260924 13:41:41.187070 13796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:41.187091 13801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15def8b08c784724b55300d4c13e5e0d" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39771 } }
I20260924 13:41:41.187484 13801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15def8b08c784724b55300d4c13e5e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15def8b08c784724b55300d4c13e5e0d" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39771 } } }
I20260924 13:41:41.187579 13801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:41.187502 13802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15def8b08c784724b55300d4c13e5e0d. Latest consensus state: current_term: 1 leader_uuid: "15def8b08c784724b55300d4c13e5e0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15def8b08c784724b55300d4c13e5e0d" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 39771 } } }
I20260924 13:41:41.187736 13802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:41.187881 13809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:41.188382 13809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:41.190003 13809 catalog_manager.cc:1347] Generated new cluster ID: d0e5323161d240528b49cf2afa9db2b0
I20260924 13:41:41.190053 13809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:41.191963 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.186279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.25.240.126:39771)
0924 13:41:41.186527 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.186537 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.186572 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:41:41.186687 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.186694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.186770 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.186974 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:41.187205 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.188827 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.189842 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.190434 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.190555 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.191289 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:41.191293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:41.191293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:41.191497 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:41.191534 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:41.191548 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:41.191610 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.191706 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.191777 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260924 13:41:41.213701 13809 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:41.214499 13809 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:41.224903 13809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d: Generated new TSK 0
I20260924 13:41:41.225109 13809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:41.232234 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.65:0
--local_ip_for_outbound_sockets=127.25.240.65
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39771
--builtin_ntp_servers=127.25.240.84:37897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:41.312408 13820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:41.312611 13820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:41.312633 13820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:41.313794 13820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:41.313833 13820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:41.313884 13820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.65
I20260924 13:41:41.315232 13820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.240.65
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.65
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:41.315470 13820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:41.315791 13820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:41.320514 13826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:41.320528 13825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:41.320530 13828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:41.320788 13820 server_base.cc:1034] running on GCE node
I20260924 13:41:41.321002 13820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:41.321194 13820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:41.322326 13820 hybrid_clock.cc:648] HybridClock initialized: now 1790257301322309 us; error 31 us; skew 500 ppm
I20260924 13:41:41.323570 13820 webserver.cc:460] Webserver started at http://127.25.240.65:37999/ using document root <none> and password file <none>
I20260924 13:41:41.323771 13820 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:41.323810 13820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:41.323897 13820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:41.324867 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/data/instance:
uuid: "15bbb4bfecca40279d7498aef8c4af11"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "fd1a710321bca283e8287d878486a83a"
server_key_iv: "c835e32f27326e6a6c80a33f70c6e26b"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.325165 13820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance:
uuid: "15bbb4bfecca40279d7498aef8c4af11"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "fd1a710321bca283e8287d878486a83a"
server_key_iv: "c835e32f27326e6a6c80a33f70c6e26b"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.326386 13820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:41.326997 13834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.327158 13820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:41:41.327243 13820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/wal
uuid: "15bbb4bfecca40279d7498aef8c4af11"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "fd1a710321bca283e8287d878486a83a"
server_key_iv: "c835e32f27326e6a6c80a33f70c6e26b"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.327317 13820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:41.343576 13820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:41.343874 13820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:41.344033 13820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:41.344250 13820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:41.344563 13820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:41.344612 13820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.344651 13820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:41.344676 13820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.350369 13820 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.65:42957
I20260924 13:41:41.350419 13947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.65:42957 every 8 connection(s)
I20260924 13:41:41.350724 13820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/data/info.pb
I20260924 13:41:41.355826 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.351351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.65:53483 (local address 127.25.240.126:39771)
0924 13:41:41.351398 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.351400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.351639 (+ 239us) server_negotiation.cc:407] Connection header received
0924 13:41:41.352166 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.352167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.352182 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.352246 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:41.352894 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.353403 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.354418 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.354644 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.354730 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.355551 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:41.355553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:41.355553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:41.355557 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:41.355570 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:41.355574 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:41.355627 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.355714 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.355746 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 13:41:41.356040 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.351355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39771 (local address 127.25.240.65:53483)
0924 13:41:41.351520 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.351551 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:41.352051 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:41.352315 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:41.352319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:41.352327 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:41.352799 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.352803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.353522 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.353523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.354325 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.354327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.354738 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.354739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.354872 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.355401 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:41.355402 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:41.355430 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:41.355451 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:41.355630 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:41.355634 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:41.355638 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:41.355718 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 13:41:41.355813 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260924 13:41:41.356550 13948 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39771
I20260924 13:41:41.356638 13948 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:41.356814 13948 heartbeater.cc:511] Master 127.25.240.126:39771 requested a full tablet report, sending...
I20260924 13:41:41.357213 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 13820
I20260924 13:41:41.357317 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-0/wal/instance
I20260924 13:41:41.357478 13760 ts_manager.cc:194] Registered new tserver with Master: 15bbb4bfecca40279d7498aef8c4af11 (127.25.240.65:42957)
I20260924 13:41:41.357506 26561 external_mini_cluster.cc:1428] Setting key d7305b290b9688a9c20257adaeac8210
I20260924 13:41:41.358165 13760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.65:53483
I20260924 13:41:41.358695 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.66:0
--local_ip_for_outbound_sockets=127.25.240.66
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39771
--builtin_ntp_servers=127.25.240.84:37897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:41.439237 13951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:41.439466 13951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:41.439502 13951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:41.440611 13951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:41.440668 13951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:41.440709 13951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.66
I20260924 13:41:41.442051 13951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.240.66
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.66
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:41.442304 13951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:41.442589 13951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:41.447440 13956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:41.447578 13951 server_base.cc:1034] running on GCE node
W20260924 13:41:41.447432 13959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:41.447433 13957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:41.448015 13951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:41.448257 13951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:41.449409 13951 hybrid_clock.cc:648] HybridClock initialized: now 1790257301449399 us; error 41 us; skew 500 ppm
I20260924 13:41:41.450623 13951 webserver.cc:460] Webserver started at http://127.25.240.66:44935/ using document root <none> and password file <none>
I20260924 13:41:41.450863 13951 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:41.450935 13951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:41.451069 13951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:41.452124 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/data/instance:
uuid: "b00d822f17a141b2b1448e7975004bc6"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "10d8e830e5466691b4717f4134133ad4"
server_key_iv: "30d988415d147cb2c71fe3f07360d842"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.452482 13951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance:
uuid: "b00d822f17a141b2b1448e7975004bc6"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "10d8e830e5466691b4717f4134133ad4"
server_key_iv: "30d988415d147cb2c71fe3f07360d842"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.453866 13951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:41.454583 13965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.454768 13951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:41.454849 13951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/wal
uuid: "b00d822f17a141b2b1448e7975004bc6"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "10d8e830e5466691b4717f4134133ad4"
server_key_iv: "30d988415d147cb2c71fe3f07360d842"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.454901 13951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:41.469453 13951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:41.469794 13951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:41.469944 13951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:41.470155 13951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:41.470484 13951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:41.470515 13951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.470556 13951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:41.470582 13951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.476294 13951 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.66:44231
I20260924 13:41:41.476354 14078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.66:44231 every 8 connection(s)
I20260924 13:41:41.476650 13951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/data/info.pb
I20260924 13:41:41.481455 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.477334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.66:54629 (local address 127.25.240.126:39771)
0924 13:41:41.477419 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.477422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.477718 (+ 296us) server_negotiation.cc:407] Connection header received
0924 13:41:41.478222 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.478223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.478233 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.478286 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:41.478888 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.479379 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.480317 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.480507 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.480561 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.481253 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:41.481254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:41.481255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:41.481259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:41.481271 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:41.481276 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:41.481321 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.481376 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.481397 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 13:41:41.481720 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.477319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39771 (local address 127.25.240.66:54629)
0924 13:41:41.477600 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.477635 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:41.478135 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:41.478337 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:41.478340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:41.478348 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:41.478824 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.478828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.479453 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.479454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.480246 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.480248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.480577 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.480578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.480681 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.481136 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:41.481137 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:41.481163 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:41.481184 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:41.481324 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:41.481326 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:41.481328 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 13:41:41.481373 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 13:41:41.481445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":125,"threads_started":1}
I20260924 13:41:41.482178 14079 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39771
I20260924 13:41:41.482264 14079 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:41.482403 14079 heartbeater.cc:511] Master 127.25.240.126:39771 requested a full tablet report, sending...
I20260924 13:41:41.482709 13760 ts_manager.cc:194] Registered new tserver with Master: b00d822f17a141b2b1448e7975004bc6 (127.25.240.66:44231)
I20260924 13:41:41.483059 13760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.66:54629
I20260924 13:41:41.483287 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 13951
I20260924 13:41:41.483392 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/wal/instance
I20260924 13:41:41.483572 26561 external_mini_cluster.cc:1428] Setting key 3af2c21acf6c4cbb9e5b556b1e3910fe
I20260924 13:41:41.484388 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.240.67:0
--local_ip_for_outbound_sockets=127.25.240.67
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39771
--builtin_ntp_servers=127.25.240.84:37897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 13:41:41.565214 14082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:41.565419 14082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:41.565440 14082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:41.566519 14082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:41.566551 14082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:41.566576 14082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.240.67
I20260924 13:41:41.567803 14082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:37897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.240.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.240.67
--webserver_port=0
--tserver_master_addrs=127.25.240.126:39771
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.240.67
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:41.567979 14082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:41.568207 14082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:41.573410 14088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:41.573534 14087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:41.573601 14082 server_base.cc:1034] running on GCE node
W20260924 13:41:41.573726 14090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:41.573985 14082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:41.574226 14082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:41.575380 14082 hybrid_clock.cc:648] HybridClock initialized: now 1790257301575375 us; error 38 us; skew 500 ppm
I20260924 13:41:41.576606 14082 webserver.cc:460] Webserver started at http://127.25.240.67:33019/ using document root <none> and password file <none>
I20260924 13:41:41.576851 14082 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:41.576922 14082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:41.577059 14082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:41.578181 14082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/data/instance:
uuid: "3c20c8b3b2c048a88f02c78263f533b3"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "2942c0c2c962715a4970cfb8522a4ae0"
server_key_iv: "71908751d4e61b721164f6b4d81aef93"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.578567 14082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance:
uuid: "3c20c8b3b2c048a88f02c78263f533b3"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "2942c0c2c962715a4970cfb8522a4ae0"
server_key_iv: "71908751d4e61b721164f6b4d81aef93"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.579841 14082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 13:41:41.580580 14096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.580758 14082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:41.580858 14082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/wal
uuid: "3c20c8b3b2c048a88f02c78263f533b3"
format_stamp: "Formatted at 2026-09-24 13:41:41 on dist-test-slave-m5z1"
server_key: "2942c0c2c962715a4970cfb8522a4ae0"
server_key_iv: "71908751d4e61b721164f6b4d81aef93"
server_key_version: "encryptionkey@0"
I20260924 13:41:41.580967 14082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:41.602669 14082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:41.602990 14082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:41.603137 14082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:41.603364 14082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 13:41:41.603700 14082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 13:41:41.603756 14082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.603792 14082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 13:41:41.603818 14082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:41.609411 14082 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.67:41327
I20260924 13:41:41.609463 14209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.67:41327 every 8 connection(s)
I20260924 13:41:41.609849 14082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/data/info.pb
I20260924 13:41:41.615950 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.610624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:37051 (local address 127.25.240.126:39771)
0924 13:41:41.610675 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.610678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.611758 (+ 1080us) server_negotiation.cc:407] Connection header received
0924 13:41:41.612287 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.612288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.612299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.612362 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:41.613055 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.613637 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.614618 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.614839 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.614922 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.615675 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:41.615676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:41.615677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:41.615682 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:41.615696 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:41.615700 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:41.615754 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.615840 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.615869 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:41:41.616218 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.610625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:39771 (local address 127.25.240.67:37051)
0924 13:41:41.611630 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.611665 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:41.612181 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:41.612433 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:41.612438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:41.612448 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:41:41.612953 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.612958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.613732 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.613733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.614518 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.614520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.614937 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.614938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.615065 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.615532 (+ 467us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 13:41:41.615532 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:41:41.615558 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 13:41:41.615579 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:41:41.615757 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:41:41.615761 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:41:41.615764 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:41.615842 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 13:41:41.615980 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":97,"threads_started":1}
I20260924 13:41:41.616695 14210 heartbeater.cc:347] Connected to a master server at 127.25.240.126:39771
I20260924 13:41:41.616786 14210 heartbeater.cc:464] Registering TS with master...
I20260924 13:41:41.616931 14210 heartbeater.cc:511] Master 127.25.240.126:39771 requested a full tablet report, sending...
I20260924 13:41:41.617268 13760 ts_manager.cc:194] Registered new tserver with Master: 3c20c8b3b2c048a88f02c78263f533b3 (127.25.240.67:41327)
I20260924 13:41:41.617691 13760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.240.67:37051
I20260924 13:41:41.619047 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 14082
I20260924 13:41:41.619129 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/wal/instance
I20260924 13:41:41.619264 26561 external_mini_cluster.cc:1428] Setting key 0368eae8e3485b70635ae592780060ca
I20260924 13:41:41.620117 26561 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 13:41:41.623931 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.621430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.25.240.126:39771)
0924 13:41:41.621468 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.621470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.621525 (+ 55us) server_negotiation.cc:407] Connection header received
0924 13:41:41.621638 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.621638 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.621654 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.621698 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:41.621836 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.622272 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.622952 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.623117 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.623164 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.623781 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:41.623781 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:41.623782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:41.623785 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:41.623793 (+ 8us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:41.623796 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:41.623833 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.623867 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.623883 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:41:41.625244 13760 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 13:41:41.626734 13760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 13:41:41.632862 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.628948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.25.240.65:42957)
0924 13:41:41.629114 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.629117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.629170 (+ 53us) server_negotiation.cc:407] Connection header received
0924 13:41:41.629242 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.629244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.629258 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.629338 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:41.629578 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.630397 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.632003 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.632470 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.632549 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.632638 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.632744 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.632779 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260924 13:41:41.632903 14218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.628679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:44231 (local address 127.0.0.1:52618)
0924 13:41:41.628821 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.628860 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:41.628920 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:41.629498 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:41.629498 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:41.629500 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:41.629533 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.629534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.630865 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.630866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.631644 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.631646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.632591 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.632592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.632654 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.632662 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 13:41:41.632705 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 13:41:41.632764 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1,"thread_start_us":50,"threads_started":1}
I20260924 13:41:41.632884 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.628711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:42957 (local address 127.0.0.1:47914)
0924 13:41:41.629119 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.629128 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:41.629176 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:41.629422 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:41.629423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:41.629426 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:41.629475 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.629477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.630475 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.630476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.631931 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.631933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.632571 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.632572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.632647 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.632660 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 13:41:41.632730 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:41:41.632777 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":93,"threads_started":1}
I20260924 13:41:41.633056 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.628900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49144 (local address 127.25.240.67:41327)
0924 13:41:41.629066 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.629071 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.629147 (+ 76us) server_negotiation.cc:407] Connection header received
0924 13:41:41.629241 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.629243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.629258 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.629338 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:41.629524 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.630630 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.631861 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.632516 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.632576 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.632656 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.632924 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.632967 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260924 13:41:41.633183 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.628711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.67:41327 (local address 127.0.0.1:49144)
0924 13:41:41.629080 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.629091 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:41.629176 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:41.629367 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:41.629369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:41.629371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:41.629441 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.629444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.630708 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.630709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.631788 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.631791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.632688 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.632689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.632739 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.632742 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:41.632785 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 13:41:41.632823 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":25,"thread_start_us":93,"threads_started":1}
I20260924 13:41:41.633436 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.628771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.25.240.66:44231)
0924 13:41:41.629322 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.629326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.629337 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:41:41.629374 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.629376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.629390 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.629627 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:41.629689 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.630786 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.631737 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.632484 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.633193 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.633293 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.633320 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.633359 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":91,"threads_started":1}
I20260924 13:41:41.633829 13882 tablet_service.cc:1467] Processing CreateTablet for tablet 04471a2e81d74d70b82b4fd2e73d462c (DEFAULT_TABLE table=test-table [id=5324b2b03b8a4e2e8da81b3e451e1aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:41.634083 14013 tablet_service.cc:1467] Processing CreateTablet for tablet 04471a2e81d74d70b82b4fd2e73d462c (DEFAULT_TABLE table=test-table [id=5324b2b03b8a4e2e8da81b3e451e1aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:41.634202 13882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04471a2e81d74d70b82b4fd2e73d462c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:41.634387 14013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04471a2e81d74d70b82b4fd2e73d462c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:41.637557 14224 tablet_bootstrap.cc:492] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11: Bootstrap starting.
I20260924 13:41:41.637878 14225 tablet_bootstrap.cc:492] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Bootstrap starting.
I20260924 13:41:41.637938 14144 tablet_service.cc:1467] Processing CreateTablet for tablet 04471a2e81d74d70b82b4fd2e73d462c (DEFAULT_TABLE table=test-table [id=5324b2b03b8a4e2e8da81b3e451e1aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 13:41:41.638273 14224 tablet_bootstrap.cc:654] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:41.638281 14144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04471a2e81d74d70b82b4fd2e73d462c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:41.638506 14225 tablet_bootstrap.cc:654] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:41.638607 14224 log.cc:824] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:41.638814 14225 log.cc:824] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:41.639421 14224 tablet_bootstrap.cc:492] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11: No bootstrap required, opened a new log
I20260924 13:41:41.639496 14224 ts_tablet_manager.cc:1397] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 13:41:41.640934 14228 tablet_bootstrap.cc:492] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: Bootstrap starting.
I20260924 13:41:41.641350 14224 raft_consensus.cc:348] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.641513 14224 raft_consensus.cc:374] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:41.641528 14228 tablet_bootstrap.cc:654] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:41.641556 14224 raft_consensus.cc:729] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15bbb4bfecca40279d7498aef8c4af11, State: Initialized, Role: FOLLOWER
I20260924 13:41:41.641639 14225 tablet_bootstrap.cc:492] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: No bootstrap required, opened a new log
I20260924 13:41:41.641713 14225 ts_tablet_manager.cc:1397] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 13:41:41.641698 14224 consensus_queue.cc:260] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.641862 14228 log.cc:824] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:41.641978 14224 ts_tablet_manager.cc:1428] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 13:41:41.642125 13948 heartbeater.cc:503] Master 127.25.240.126:39771 was elected leader, sending a full tablet report...
I20260924 13:41:41.642793 14228 tablet_bootstrap.cc:492] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: No bootstrap required, opened a new log
I20260924 13:41:41.642863 14228 ts_tablet_manager.cc:1397] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 13:41:41.644346 14225 raft_consensus.cc:348] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.644493 14225 raft_consensus.cc:374] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:41.644515 14225 raft_consensus.cc:729] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b00d822f17a141b2b1448e7975004bc6, State: Initialized, Role: FOLLOWER
I20260924 13:41:41.644644 14225 consensus_queue.cc:260] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.644894 14225 ts_tablet_manager.cc:1428] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 13:41:41.644908 14079 heartbeater.cc:503] Master 127.25.240.126:39771 was elected leader, sending a full tablet report...
I20260924 13:41:41.644853 14228 raft_consensus.cc:348] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.645001 14228 raft_consensus.cc:374] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:41.645042 14228 raft_consensus.cc:729] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c20c8b3b2c048a88f02c78263f533b3, State: Initialized, Role: FOLLOWER
I20260924 13:41:41.645149 14228 consensus_queue.cc:260] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.645388 14228 ts_tablet_manager.cc:1428] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 13:41:41.645514 14210 heartbeater.cc:503] Master 127.25.240.126:39771 was elected leader, sending a full tablet report...
I20260924 13:41:41.705605 14232 raft_consensus.cc:482] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 13:41:41.705736 14232 raft_consensus.cc:504] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.706126 14232 leader_election.cc:288] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b00d822f17a141b2b1448e7975004bc6 (127.25.240.66:44231), 15bbb4bfecca40279d7498aef8c4af11 (127.25.240.65:42957)
I20260924 13:41:41.709983 14223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.706482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:33843 (local address 127.25.240.65:42957)
0924 13:41:41.706526 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.706527 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.706573 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:41:41.706616 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.706616 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.706625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.706697 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:41.706806 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.707490 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.708944 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.709616 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.709689 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.709698 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.709861 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.709911 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 13:41:41.710007 14221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.706319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.240.67:41707 (local address 127.25.240.66:44231)
0924 13:41:41.706365 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.706367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.706376 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:41:41.706410 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.706411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.706420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.706479 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 13:41:41.706672 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.707443 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.708944 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.709596 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.709693 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.709702 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.709893 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.709940 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:41:41.710013 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.706248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.66:44231 (local address 127.25.240.67:41707)
0924 13:41:41.706299 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.706311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:41.706353 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:41.706517 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:41.706517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:41.706519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:41.706595 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.706597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.707547 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.707548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.708827 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.708830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.709695 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.709696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.709774 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.709778 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 13:41:41.709836 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 13:41:41.709891 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":49}
I20260924 13:41:41.710023 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.706256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:42957 (local address 127.25.240.67:33843)
0924 13:41:41.706504 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.706513 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:41:41.706565 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:41:41.706703 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:41:41.706704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:41:41.706705 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 13:41:41.706747 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.706747 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.707561 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.707561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.708827 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:41:41.708830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.709696 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.709697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:41:41.709782 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.709785 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 13:41:41.709836 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 13:41:41.709890 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":85,"thread_start_us":92,"threads_started":1}
I20260924 13:41:41.710459 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04471a2e81d74d70b82b4fd2e73d462c" candidate_uuid: "3c20c8b3b2c048a88f02c78263f533b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bbb4bfecca40279d7498aef8c4af11" is_pre_election: true
I20260924 13:41:41.710479 14033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04471a2e81d74d70b82b4fd2e73d462c" candidate_uuid: "3c20c8b3b2c048a88f02c78263f533b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00d822f17a141b2b1448e7975004bc6" is_pre_election: true
I20260924 13:41:41.710629 14033 raft_consensus.cc:2454] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c20c8b3b2c048a88f02c78263f533b3 in term 0.
I20260924 13:41:41.710629 13902 raft_consensus.cc:2454] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c20c8b3b2c048a88f02c78263f533b3 in term 0.
I20260924 13:41:41.710870 14099 leader_election.cc:302] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bbb4bfecca40279d7498aef8c4af11, 3c20c8b3b2c048a88f02c78263f533b3; no voters:
I20260924 13:41:41.711028 14232 raft_consensus.cc:2781] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 13:41:41.711086 14232 raft_consensus.cc:482] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 13:41:41.711112 14232 raft_consensus.cc:3037] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:41.711867 14232 raft_consensus.cc:504] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.711999 14232 leader_election.cc:288] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [CANDIDATE]: Term 1 election: Requested vote from peers b00d822f17a141b2b1448e7975004bc6 (127.25.240.66:44231), 15bbb4bfecca40279d7498aef8c4af11 (127.25.240.65:42957)
I20260924 13:41:41.712137 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04471a2e81d74d70b82b4fd2e73d462c" candidate_uuid: "3c20c8b3b2c048a88f02c78263f533b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15bbb4bfecca40279d7498aef8c4af11"
I20260924 13:41:41.712174 14033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04471a2e81d74d70b82b4fd2e73d462c" candidate_uuid: "3c20c8b3b2c048a88f02c78263f533b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b00d822f17a141b2b1448e7975004bc6"
I20260924 13:41:41.712208 13902 raft_consensus.cc:3037] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:41.712287 14033 raft_consensus.cc:3037] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:41.712939 13902 raft_consensus.cc:2454] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c20c8b3b2c048a88f02c78263f533b3 in term 1.
I20260924 13:41:41.713101 14099 leader_election.cc:302] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15bbb4bfecca40279d7498aef8c4af11, 3c20c8b3b2c048a88f02c78263f533b3; no voters:
I20260924 13:41:41.713165 14033 raft_consensus.cc:2454] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c20c8b3b2c048a88f02c78263f533b3 in term 1.
I20260924 13:41:41.713191 14232 raft_consensus.cc:2781] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:41.713347 14232 raft_consensus.cc:686] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c20c8b3b2c048a88f02c78263f533b3, State: Running, Role: LEADER
I20260924 13:41:41.713447 14232 consensus_queue.cc:237] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } }
I20260924 13:41:41.714967 13759 catalog_manager.cc:5495] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c20c8b3b2c048a88f02c78263f533b3 (127.25.240.67). New cstate: current_term: 1 leader_uuid: "3c20c8b3b2c048a88f02c78263f533b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c20c8b3b2c048a88f02c78263f533b3" member_type: VOTER last_known_addr { host: "127.25.240.67" port: 41327 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 } health_report { overall_health: UNKNOWN } } }
W20260924 13:41:41.727151 14080 tablet.cc:2369] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 13:41:41.728793 14222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.726638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.25.240.67:41327)
0924 13:41:41.726679 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:41.726681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:41.726689 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:41:41.726739 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:41.726739 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:41.726756 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:41.726815 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:41:41.726906 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.727411 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.728135 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:41.728314 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:41.728367 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:41.728487 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:41:41.728579 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:41:41.728638 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:41.728666 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 13:41:41.728681 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:41:41.730497 14033 raft_consensus.cc:1261] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 1 FOLLOWER]: Refusing update from remote peer 3c20c8b3b2c048a88f02c78263f533b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:41.730540 13902 raft_consensus.cc:1261] T 04471a2e81d74d70b82b4fd2e73d462c P 15bbb4bfecca40279d7498aef8c4af11 [term 1 FOLLOWER]: Refusing update from remote peer 3c20c8b3b2c048a88f02c78263f533b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:41:41.730872 14234 consensus_queue.cc:1035] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b00d822f17a141b2b1448e7975004bc6" member_type: VOTER last_known_addr { host: "127.25.240.66" port: 44231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:41.731004 14232 consensus_queue.cc:1035] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15bbb4bfecca40279d7498aef8c4af11" member_type: VOTER last_known_addr { host: "127.25.240.65" port: 42957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:41:41.733422 14238 mvcc.cc:205] Tried to move back new op lower bound from 7332893907885719552 to 7332893907819110400. Current Snapshot: MvccSnapshot[applied={T|T < 7332893907885719552}]
I20260924 13:41:41.733664 14239 mvcc.cc:205] Tried to move back new op lower bound from 7332893907885719552 to 7332893907819110400. Current Snapshot: MvccSnapshot[applied={T|T < 7332893907885719552}]
I20260924 13:41:41.735924 13759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:41.736016 13759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 13:41:41.736713 13759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15def8b08c784724b55300d4c13e5e0d: Sending DeleteTablet for 3 replicas of tablet 04471a2e81d74d70b82b4fd2e73d462c
I20260924 13:41:41.737044 14144 tablet_service.cc:1514] Processing DeleteTablet for tablet 04471a2e81d74d70b82b4fd2e73d462c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:41 UTC) from {username='slave'} at 127.0.0.1:49144
I20260924 13:41:41.737092 13882 tablet_service.cc:1514] Processing DeleteTablet for tablet 04471a2e81d74d70b82b4fd2e73d462c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:41 UTC) from {username='slave'} at 127.0.0.1:47914
I20260924 13:41:41.737253 14247 tablet_replica.cc:330] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: stopping tablet replica
I20260924 13:41:41.737293 14013 tablet_service.cc:1514] Processing DeleteTablet for tablet 04471a2e81d74d70b82b4fd2e73d462c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 13:41:41 UTC) from {username='slave'} at 127.0.0.1:52618
I20260924 13:41:41.737349 14247 raft_consensus.cc:2229] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 13:41:41.737489 14247 raft_consensus.cc:2258] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:41.737708 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 13820
I20260924 13:41:41.738524 14247 ts_tablet_manager.cc:1905] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:41.740604 14247 ts_tablet_manager.cc:1918] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:41.740680 14247 log.cc:1196] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-2/wal/wals/04471a2e81d74d70b82b4fd2e73d462c
I20260924 13:41:41.740940 14247 ts_tablet_manager.cc:1939] T 04471a2e81d74d70b82b4fd2e73d462c P 3c20c8b3b2c048a88f02c78263f533b3: Deleting consensus metadata
I20260924 13:41:41.741153 14249 tablet_replica.cc:330] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: stopping tablet replica
I20260924 13:41:41.741253 14249 raft_consensus.cc:2229] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 13:41:41.741353 14249 raft_consensus.cc:2258] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 13:41:41.741371 13744 catalog_manager.cc:4841] TS 3c20c8b3b2c048a88f02c78263f533b3 (127.25.240.67:41327): tablet 04471a2e81d74d70b82b4fd2e73d462c (table test-table [id=5324b2b03b8a4e2e8da81b3e451e1aab]) successfully deleted
I20260924 13:41:41.741822 14249 ts_tablet_manager.cc:1905] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 13:41:41.743876 14249 ts_tablet_manager.cc:1918] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 13:41:41.743996 14249 log.cc:1196] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Deleting WAL directory at /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790257062855231-26561-0/minicluster-data/ts-1/wal/wals/04471a2e81d74d70b82b4fd2e73d462c
I20260924 13:41:41.744270 14249 ts_tablet_manager.cc:1939] T 04471a2e81d74d70b82b4fd2e73d462c P b00d822f17a141b2b1448e7975004bc6: Deleting consensus metadata
I20260924 13:41:41.744683 13747 catalog_manager.cc:4841] TS b00d822f17a141b2b1448e7975004bc6 (127.25.240.66:44231): tablet 04471a2e81d74d70b82b4fd2e73d462c (table test-table [id=5324b2b03b8a4e2e8da81b3e451e1aab]) successfully deleted
W20260924 13:41:41.744753 13746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.240.65:42957 (error 108)
I20260924 13:41:41.745024 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 13951
I20260924 13:41:41.745549 14220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:41.745089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.65:42957 (local address 127.0.0.1:47930)
0924 13:41:41.745469 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 13:41:41.745511 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.65:42957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
W20260924 13:41:41.750138 13746 catalog_manager.cc:4584] TS 15bbb4bfecca40279d7498aef8c4af11 (127.25.240.65:42957): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04471a2e81d74d70b82b4fd2e73d462c: Network error: Client connection negotiation failed: client connection to 127.25.240.65:42957: connect: Connection refused (error 111)
I20260924 13:41:41.750609 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 14082
I20260924 13:41:41.755004 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 13729
2026-09-24T13:41:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (719 ms)
[----------] 6 tests from EncryptionPolicyTest (8091 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:41 dist-test-slave-m5z1 krb5kdc[14253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:41 dist-test-slave-m5z1 krb5kdc[14253](info): set up 2 sockets
Sep 24 13:41:41 dist-test-slave-m5z1 krb5kdc[14253](info): commencing operation
krb5kdc: starting...
W20260924 13:41:42.802089 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:42 dist-test-slave-m5z1 krb5kdc[14253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:42.931743 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:43.010866 14266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:43.011106 14266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:43.011144 14266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:43.012291 14266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:43.012370 14266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:43.012396 14266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:43.012428 14266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:43.012455 14266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:43.013911 14266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:43.014173 14266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:43.014395 14266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:43.019124 14272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:43.019114 14271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:43.019115 14274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:43.019661 14266 server_base.cc:1034] running on GCE node
I20260924 13:41:43.019919 14266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:43.020201 14266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:43.021358 14266 hybrid_clock.cc:648] HybridClock initialized: now 1790257303021348 us; error 36 us; skew 500 ppm
Sep 24 13:41:43 dist-test-slave-m5z1 krb5kdc[14253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:43.023737 14266 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:43.024137 14266 webserver.cc:460] Webserver started at http://127.0.0.1:33435/ using document root <none> and password file <none>
I20260924 13:41:43.024348 14266 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:43.024427 14266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:43.024564 14266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:43.025620 14266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "9f30a99fd434418c9a6f0f99ea9996d8"
format_stamp: "Formatted at 2026-09-24 13:41:43 on dist-test-slave-m5z1"
server_key: "aa8135058f0780db641e9e40ea23b9b8"
server_key_iv: "8665dd98eb04d0dce124b1cf32772465"
server_key_version: "encryptionkey@0"
I20260924 13:41:43.025985 14266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "9f30a99fd434418c9a6f0f99ea9996d8"
format_stamp: "Formatted at 2026-09-24 13:41:43 on dist-test-slave-m5z1"
server_key: "aa8135058f0780db641e9e40ea23b9b8"
server_key_iv: "8665dd98eb04d0dce124b1cf32772465"
server_key_version: "encryptionkey@0"
I20260924 13:41:43.027254 14266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:41:43.028005 14281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:43.028177 14266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:41:43.028268 14266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "9f30a99fd434418c9a6f0f99ea9996d8"
format_stamp: "Formatted at 2026-09-24 13:41:43 on dist-test-slave-m5z1"
server_key: "aa8135058f0780db641e9e40ea23b9b8"
server_key_iv: "8665dd98eb04d0dce124b1cf32772465"
server_key_version: "encryptionkey@0"
I20260924 13:41:43.028345 14266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:43.035292 14266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:43.037441 14266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:43.037613 14266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:43.041170 14266 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35739
I20260924 13:41:43.041234 14333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35739 every 8 connection(s)
I20260924 13:41:43.041630 14266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:43.042469 14334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:43.044678 14334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8: Bootstrap starting.
I20260924 13:41:43.045230 14334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:43.045506 14334 log.cc:824] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:43.045989 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 14266
I20260924 13:41:43.046097 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:43.046290 26561 external_mini_cluster.cc:1428] Setting key 80ab1f2fa52daaf14e34b46ac0099392
I20260924 13:41:43.046326 14334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8: No bootstrap required, opened a new log
I20260924 13:41:43.048686 14334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f30a99fd434418c9a6f0f99ea9996d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35739 } }
I20260924 13:41:43.048811 14334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:43.048833 14334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f30a99fd434418c9a6f0f99ea9996d8, State: Initialized, Role: FOLLOWER
I20260924 13:41:43.048980 14334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f30a99fd434418c9a6f0f99ea9996d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35739 } }
I20260924 13:41:43.049054 14334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:43.049077 14334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:43.049147 14334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:43.049738 14334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f30a99fd434418c9a6f0f99ea9996d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35739 } }
I20260924 13:41:43.049836 14334 leader_election.cc:302] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f30a99fd434418c9a6f0f99ea9996d8; no voters:
I20260924 13:41:43.049990 14334 leader_election.cc:288] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:43.050025 14339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:43.050148 14339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [term 1 LEADER]: Becoming Leader. State: Replica: 9f30a99fd434418c9a6f0f99ea9996d8, State: Running, Role: LEADER
I20260924 13:41:43.050235 14334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:43.050221 14339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f30a99fd434418c9a6f0f99ea9996d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35739 } }
I20260924 13:41:43.050527 14340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f30a99fd434418c9a6f0f99ea9996d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f30a99fd434418c9a6f0f99ea9996d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35739 } } }
I20260924 13:41:43.050643 14340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:43.050526 14341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f30a99fd434418c9a6f0f99ea9996d8. Latest consensus state: current_term: 1 leader_uuid: "9f30a99fd434418c9a6f0f99ea9996d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f30a99fd434418c9a6f0f99ea9996d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35739 } } }
I20260924 13:41:43.050835 14341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:43.051122 14345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:43.051614 14345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 13:41:43 dist-test-slave-m5z1 krb5kdc[14253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:41:43.054463 14345 catalog_manager.cc:1347] Generated new cluster ID: 693131ca6d1e470a8a787240ff0bef9b
I20260924 13:41:43.054515 14345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:43.058799 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:43.046809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44113 (local address 127.0.0.1:35739)
0924 13:41:43.046990 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:43.046997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:43.047021 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:41:43.047624 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:43.047632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:43.047692 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:43.047808 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:43.048600 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:43.050080 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:43.051118 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:43.051759 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:43.051872 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:43.054897 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:43.054902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:43.054903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:43.054915 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:43.056858 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:43.057290 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:43.057293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:43.057293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:43.057404 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:43.057646 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:43.057647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:43.057648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:43.057972 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:43.058138 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:43.058222 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 13:41:43.058447 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260924 13:41:43.060048 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:43.060817 14345 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:43.061380 14345 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:43.069058 14345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f30a99fd434418c9a6f0f99ea9996d8: Generated new TSK 0
I20260924 13:41:43.069200 14345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:43.069351 14338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:43.061683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43831 (local address 127.0.0.1:35739)
0924 13:41:43.061717 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:43.061719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:43.061783 (+ 64us) server_negotiation.cc:407] Connection header received
0924 13:41:43.061885 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:43.061887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:43.061897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:43.061927 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:43.062598 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:43.063388 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:43.064479 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:43.064788 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:43.064827 (+ 39us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:43.066343 (+ 1516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:43.066344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:43.066345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:43.066352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:43.067837 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:43.068278 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:43.068279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:43.068279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:43.068342 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:43.068574 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:43.068575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:43.068575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:43.068703 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:43.068786 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:43.069073 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 13:41:43.069140 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:41:43.071408 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 14266
W20260924 13:41:43.075687 14251 connection.cc:535] client connection to 127.0.0.1:35739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35739): Connection reset by peer (error 104)
2026-09-24T13:41:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:43 dist-test-slave-m5z1 krb5kdc[14365](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:43 dist-test-slave-m5z1 krb5kdc[14365](info): set up 2 sockets
Sep 24 13:41:43 dist-test-slave-m5z1 krb5kdc[14365](info): commencing operation
krb5kdc: starting...
W20260924 13:41:44.093881 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:44 dist-test-slave-m5z1 krb5kdc[14365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:44.210711 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:44.289953 14378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:44.290189 14378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:44.290225 14378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:44.291368 14378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:44.291427 14378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:44.291450 14378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:44.291471 14378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:44.291498 14378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:44.292927 14378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39537
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:44.293200 14378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:44.293417 14378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:44.298262 14386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:44.298424 14378 server_base.cc:1034] running on GCE node
W20260924 13:41:44.298269 14384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:44.298273 14383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:44.298779 14378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:44.299062 14378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:44.300200 14378 hybrid_clock.cc:648] HybridClock initialized: now 1790257304300188 us; error 28 us; skew 500 ppm
Sep 24 13:41:44 dist-test-slave-m5z1 krb5kdc[14365](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257304, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:44.305771 14378 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:44.306192 14378 webserver.cc:460] Webserver started at http://127.0.0.1:42719/ using document root <none> and password file <none>
I20260924 13:41:44.306411 14378 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:44.306463 14378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:44.306612 14378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:44.307660 14378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "898d50ff38044571afaae7a10f5d1891"
format_stamp: "Formatted at 2026-09-24 13:41:44 on dist-test-slave-m5z1"
server_key: "94e37f5c04b9669a4039f2ba928c2234"
server_key_iv: "10220d1c0b0ff40150509990f7bef2df"
server_key_version: "encryptionkey@0"
I20260924 13:41:44.308007 14378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "898d50ff38044571afaae7a10f5d1891"
format_stamp: "Formatted at 2026-09-24 13:41:44 on dist-test-slave-m5z1"
server_key: "94e37f5c04b9669a4039f2ba928c2234"
server_key_iv: "10220d1c0b0ff40150509990f7bef2df"
server_key_version: "encryptionkey@0"
I20260924 13:41:44.309285 14378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:44.310083 14393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:44.310267 14378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:44.310353 14378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "898d50ff38044571afaae7a10f5d1891"
format_stamp: "Formatted at 2026-09-24 13:41:44 on dist-test-slave-m5z1"
server_key: "94e37f5c04b9669a4039f2ba928c2234"
server_key_iv: "10220d1c0b0ff40150509990f7bef2df"
server_key_version: "encryptionkey@0"
I20260924 13:41:44.310437 14378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:44.327845 14378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:44.329630 14378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:44.329779 14378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:44.333400 14378 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39537
I20260924 13:41:44.333467 14445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39537 every 8 connection(s)
I20260924 13:41:44.333822 14378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:44.334652 14446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:44.335081 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 14378
I20260924 13:41:44.335196 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:44.335381 26561 external_mini_cluster.cc:1428] Setting key bec955762e934cb06a13d890b8a6081e
I20260924 13:41:44.337394 14446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891: Bootstrap starting.
I20260924 13:41:44.338068 14446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:44.338371 14446 log.cc:824] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:44.339099 14446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891: No bootstrap required, opened a new log
I20260924 13:41:44.340793 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:44.335882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54075 (local address 127.0.0.1:39537)
0924 13:41:44.336053 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:44.336061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:44.336085 (+ 24us) server_negotiation.cc:407] Connection header received
0924 13:41:44.336494 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:44.336501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:44.336545 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:44.336658 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:44.336895 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:44.337902 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:44.338953 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:44.339378 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:44.339466 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:44.340166 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:44.340170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:44.340170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:44.340325 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:44.340353 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:44.340363 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:44.340441 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:44.340580 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 13:41:44.340611 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260924 13:41:44.341403 14446 raft_consensus.cc:348] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898d50ff38044571afaae7a10f5d1891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260924 13:41:44.341562 14446 raft_consensus.cc:374] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:44.341650 14446 raft_consensus.cc:729] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898d50ff38044571afaae7a10f5d1891, State: Initialized, Role: FOLLOWER
I20260924 13:41:44.341742 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:44.341789 14446 consensus_queue.cc:260] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898d50ff38044571afaae7a10f5d1891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260924 13:41:44.341881 14446 raft_consensus.cc:388] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:44.341921 14446 raft_consensus.cc:482] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:44.341969 14446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:44.342500 14446 raft_consensus.cc:504] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898d50ff38044571afaae7a10f5d1891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260924 13:41:44.342617 14446 leader_election.cc:302] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 898d50ff38044571afaae7a10f5d1891; no voters:
I20260924 13:41:44.342782 14446 leader_election.cc:288] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:44.342821 14455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:44.342990 14455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [term 1 LEADER]: Becoming Leader. State: Replica: 898d50ff38044571afaae7a10f5d1891, State: Running, Role: LEADER
I20260924 13:41:44.343019 14446 sys_catalog.cc:564] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:44.343098 14455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898d50ff38044571afaae7a10f5d1891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } }
I20260924 13:41:44.343421 14455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "898d50ff38044571afaae7a10f5d1891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898d50ff38044571afaae7a10f5d1891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260924 13:41:44.343541 14455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:44.343729 14455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 898d50ff38044571afaae7a10f5d1891. Latest consensus state: current_term: 1 leader_uuid: "898d50ff38044571afaae7a10f5d1891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "898d50ff38044571afaae7a10f5d1891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39537 } } }
I20260924 13:41:44.343797 14464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:44.343832 14455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:44.344527 14464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:44.347011 14464 catalog_manager.cc:1347] Generated new cluster ID: cec3b077026a4235aaf334389f8467d1
I20260924 13:41:44.347059 14464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:44.347363 14449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:44.343048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32833 (local address 127.0.0.1:39537)
0924 13:41:44.343118 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:44.343120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:44.344177 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 13:41:44.344248 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:44.344249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:44.344259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:44.344299 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:44.344563 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:44.345036 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:44.345985 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:44.346257 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:44.346321 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:44.347129 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:44.347130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:44.347130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:44.347135 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:44.347147 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:44.347151 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:44.347204 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:44.347263 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 13:41:44.347271 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 13:41:44.362778 14464 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:44.363472 14464 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:44.368230 14464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 898d50ff38044571afaae7a10f5d1891: Generated new TSK 0
I20260924 13:41:44.368405 14464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:44.374751 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 14378
W20260924 13:41:44.379207 14363 connection.cc:535] client connection to 127.0.0.1:39537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39537): Connection reset by peer (error 104)
2026-09-24T13:41:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:44 dist-test-slave-m5z1 krb5kdc[14477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:44 dist-test-slave-m5z1 krb5kdc[14477](info): set up 2 sockets
Sep 24 13:41:44 dist-test-slave-m5z1 krb5kdc[14477](info): commencing operation
krb5kdc: starting...
W20260924 13:41:45.410012 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:45 dist-test-slave-m5z1 krb5kdc[14477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:45.529702 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:45.611754 14490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:45.611958 14490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:45.611982 14490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:45.613209 14490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:45.613253 14490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:45.613268 14490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:45.613327 14490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:45.613349 14490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:45.614847 14490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36547
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:45.615098 14490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:45.615320 14490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:45.620105 14498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:45.620103 14495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:45.620119 14496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:45.620304 14490 server_base.cc:1034] running on GCE node
I20260924 13:41:45.620609 14490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:45.620899 14490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:45.622066 14490 hybrid_clock.cc:648] HybridClock initialized: now 1790257305622043 us; error 33 us; skew 500 ppm
Sep 24 13:41:45 dist-test-slave-m5z1 krb5kdc[14477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:45.624459 14490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:45.624913 14490 webserver.cc:460] Webserver started at http://127.0.0.1:46017/ using document root <none> and password file <none>
I20260924 13:41:45.625139 14490 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:45.625205 14490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:45.625355 14490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:45.626461 14490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "f9b2087e87944214a9f22f38d4263035"
format_stamp: "Formatted at 2026-09-24 13:41:45 on dist-test-slave-m5z1"
server_key: "6bdab287550050f8d54c32e2ee2c11ff"
server_key_iv: "5ca62a6c91b42efaf0d197bdd9b96923"
server_key_version: "encryptionkey@0"
I20260924 13:41:45.626819 14490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "f9b2087e87944214a9f22f38d4263035"
format_stamp: "Formatted at 2026-09-24 13:41:45 on dist-test-slave-m5z1"
server_key: "6bdab287550050f8d54c32e2ee2c11ff"
server_key_iv: "5ca62a6c91b42efaf0d197bdd9b96923"
server_key_version: "encryptionkey@0"
I20260924 13:41:45.628106 14490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:41:45.628798 14505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:45.628957 14490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:45.629055 14490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "f9b2087e87944214a9f22f38d4263035"
format_stamp: "Formatted at 2026-09-24 13:41:45 on dist-test-slave-m5z1"
server_key: "6bdab287550050f8d54c32e2ee2c11ff"
server_key_iv: "5ca62a6c91b42efaf0d197bdd9b96923"
server_key_version: "encryptionkey@0"
I20260924 13:41:45.629139 14490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:45.632406 14490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:45.634603 14490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:45.634761 14490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:45.638963 14490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36547
I20260924 13:41:45.638988 14560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36547 every 8 connection(s)
I20260924 13:41:45.639314 14490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:45.640244 14561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:45.642426 14561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035: Bootstrap starting.
I20260924 13:41:45.643015 14561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:45.643241 14561 log.cc:824] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:45.643855 14561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035: No bootstrap required, opened a new log
I20260924 13:41:45.644083 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 14490
I20260924 13:41:45.644213 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:45.644404 26561 external_mini_cluster.cc:1428] Setting key 41f098ad7f2a7ad2ff6618c8c4063bd5
I20260924 13:41:45.646230 14561 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b2087e87944214a9f22f38d4263035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } }
I20260924 13:41:45.646374 14561 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:45.646454 14561 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b2087e87944214a9f22f38d4263035, State: Initialized, Role: FOLLOWER
I20260924 13:41:45.646593 14561 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b2087e87944214a9f22f38d4263035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } }
I20260924 13:41:45.646698 14561 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:45.646744 14561 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:45.646791 14561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:45.647059 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:45.644907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59021 (local address 127.0.0.1:36547)
0924 13:41:45.645092 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:45.645101 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:45.645137 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:41:45.645781 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:45.645789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:45.645824 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:45.645972 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:45.646640 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:45.646644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:45.646644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:45.646648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:45.646683 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:45.646692 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:45.646769 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:45.646868 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 13:41:45.646908 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260924 13:41:45.647687 14561 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b2087e87944214a9f22f38d4263035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } }
I20260924 13:41:45.647797 14561 leader_election.cc:302] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b2087e87944214a9f22f38d4263035; no voters:
I20260924 13:41:45.647974 14561 leader_election.cc:288] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:45.647998 14566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:45.647996 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:45.648109 14566 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [term 1 LEADER]: Becoming Leader. State: Replica: f9b2087e87944214a9f22f38d4263035, State: Running, Role: LEADER
I20260924 13:41:45.648226 14561 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:45.648247 14566 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b2087e87944214a9f22f38d4263035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } }
I20260924 13:41:45.648605 14566 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b2087e87944214a9f22f38d4263035. Latest consensus state: current_term: 1 leader_uuid: "f9b2087e87944214a9f22f38d4263035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b2087e87944214a9f22f38d4263035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } } }
I20260924 13:41:45.648617 14567 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b2087e87944214a9f22f38d4263035" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b2087e87944214a9f22f38d4263035" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36547 } } }
I20260924 13:41:45.648849 14567 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:45.648773 14566 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:45.649194 14576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:45.649755 14576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:45.651196 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:45.649482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45463 (local address 127.0.0.1:36547)
0924 13:41:45.649516 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:45.649517 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:45.650163 (+ 646us) server_negotiation.cc:407] Connection header received
0924 13:41:45.650300 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:45.650301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:45.650306 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:45.650365 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:45.650977 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:45.650978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:45.650978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:45.650982 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:45.650996 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:45.651000 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:45.651061 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:45.651124 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 13:41:45.651136 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 13:41:45.651631 14576 catalog_manager.cc:1347] Generated new cluster ID: ab2cc16e03d347e4a55d7016a072d246
I20260924 13:41:45.651679 14576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:45.688095 14576 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:45.688314 14576 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:45.690389 14576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b2087e87944214a9f22f38d4263035: Generated new TSK 0
I20260924 13:41:45.690579 14576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:45.693118 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 14490
2026-09-24T13:41:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:45 dist-test-slave-m5z1 krb5kdc[14592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:45 dist-test-slave-m5z1 krb5kdc[14592](info): set up 2 sockets
Sep 24 13:41:45 dist-test-slave-m5z1 krb5kdc[14592](info): commencing operation
krb5kdc: starting...
W20260924 13:41:46.741770 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:46 dist-test-slave-m5z1 krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:46.857738 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:46.936048 14605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:46.936233 14605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:46.936255 14605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:46.937400 14605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:46.937453 14605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:46.937467 14605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:46.937479 14605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:46.937532 14605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:46.938992 14605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:46.939229 14605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:46.939458 14605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:46.944234 14613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:46.944211 14610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:46.944211 14611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:46.944419 14605 server_base.cc:1034] running on GCE node
I20260924 13:41:46.944742 14605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:46.945019 14605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:46.946183 14605 hybrid_clock.cc:648] HybridClock initialized: now 1790257306946173 us; error 29 us; skew 500 ppm
Sep 24 13:41:46 dist-test-slave-m5z1 krb5kdc[14592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:46.948642 14605 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:46.949043 14605 webserver.cc:460] Webserver started at http://127.0.0.1:40803/ using document root <none> and password file <none>
I20260924 13:41:46.949280 14605 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:46.949345 14605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:46.949487 14605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:46.950543 14605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "d5ca81dfd4be4c12acf0dcc3b790944f"
format_stamp: "Formatted at 2026-09-24 13:41:46 on dist-test-slave-m5z1"
server_key: "5b66fa251eec8532a111aa18d3f655af"
server_key_iv: "1b4ec1e7a05069958523c2eab0b08687"
server_key_version: "encryptionkey@0"
I20260924 13:41:46.950888 14605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "d5ca81dfd4be4c12acf0dcc3b790944f"
format_stamp: "Formatted at 2026-09-24 13:41:46 on dist-test-slave-m5z1"
server_key: "5b66fa251eec8532a111aa18d3f655af"
server_key_iv: "1b4ec1e7a05069958523c2eab0b08687"
server_key_version: "encryptionkey@0"
I20260924 13:41:46.952222 14605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:41:46.952934 14620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:46.953220 14605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:46.953320 14605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "d5ca81dfd4be4c12acf0dcc3b790944f"
format_stamp: "Formatted at 2026-09-24 13:41:46 on dist-test-slave-m5z1"
server_key: "5b66fa251eec8532a111aa18d3f655af"
server_key_iv: "1b4ec1e7a05069958523c2eab0b08687"
server_key_version: "encryptionkey@0"
I20260924 13:41:46.953390 14605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:46.962116 14605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:46.964272 14605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:46.964424 14605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:46.968355 14605 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42381
I20260924 13:41:46.968398 14672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42381 every 8 connection(s)
I20260924 13:41:46.968905 14605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:46.969666 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:46.971889 14673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f: Bootstrap starting.
I20260924 13:41:46.971959 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 14605
I20260924 13:41:46.972069 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:46.972283 26561 external_mini_cluster.cc:1428] Setting key 714cd00f34c6af188b3b8032f9dc7f85
I20260924 13:41:46.972584 14673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:46.972918 14673 log.cc:824] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:46.973654 14673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f: No bootstrap required, opened a new log
I20260924 13:41:46.975481 14673 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ca81dfd4be4c12acf0dcc3b790944f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } }
I20260924 13:41:46.975638 14673 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:46.975687 14673 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ca81dfd4be4c12acf0dcc3b790944f, State: Initialized, Role: FOLLOWER
I20260924 13:41:46.975814 14673 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ca81dfd4be4c12acf0dcc3b790944f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } }
I20260924 13:41:46.975912 14673 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:46.975955 14673 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:46.976004 14673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:46.976545 14673 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ca81dfd4be4c12acf0dcc3b790944f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } }
I20260924 13:41:46.976661 14673 leader_election.cc:302] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ca81dfd4be4c12acf0dcc3b790944f; no voters:
I20260924 13:41:46.976845 14673 leader_election.cc:288] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:46.976862 14678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:46.976984 14678 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [term 1 LEADER]: Becoming Leader. State: Replica: d5ca81dfd4be4c12acf0dcc3b790944f, State: Running, Role: LEADER
I20260924 13:41:46.977118 14673 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:46.977118 14678 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ca81dfd4be4c12acf0dcc3b790944f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } }
I20260924 13:41:46.977578 14678 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ca81dfd4be4c12acf0dcc3b790944f. Latest consensus state: current_term: 1 leader_uuid: "d5ca81dfd4be4c12acf0dcc3b790944f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ca81dfd4be4c12acf0dcc3b790944f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } } }
I20260924 13:41:46.977830 14678 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:46.977751 14680 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5ca81dfd4be4c12acf0dcc3b790944f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ca81dfd4be4c12acf0dcc3b790944f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42381 } } }
I20260924 13:41:46.978223 14680 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f [sys.catalog]: This master's current role is: LEADER
Sep 24 13:41:46 dist-test-slave-m5z1 krb5kdc[14592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260924 13:41:46.979045 14694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:41:46.979103 14694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:41:46.979526 14687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:46.979766 14687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:46.981294 14687 catalog_manager.cc:1347] Generated new cluster ID: 4b1a5fe3cd3140008422b8d6cd5b047b
I20260924 13:41:46.981344 14687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:46.983657 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:46.972779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33839 (local address 127.0.0.1:42381)
0924 13:41:46.972968 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:46.972993 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:46.973039 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:41:46.973480 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:46.973487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:46.973530 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:46.973722 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:46.974366 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:46.975260 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:46.976182 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:46.976594 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:46.976709 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:46.979503 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:46.979507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:46.979508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:46.979520 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:46.981593 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:46.982009 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:46.982012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:46.982012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:46.982114 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:46.982321 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:46.982322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:46.982322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:46.982635 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:46.982788 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:46.983287 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 13:41:46.983421 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260924 13:41:46.984540 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:46.993180 14687 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:46.993474 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:46.986119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40339 (local address 127.0.0.1:42381)
0924 13:41:46.986156 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:46.986158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:46.986223 (+ 65us) server_negotiation.cc:407] Connection header received
0924 13:41:46.986316 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:46.986317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:46.986326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:46.986357 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:46.987040 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:46.987771 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:46.988817 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:46.989121 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:46.989162 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:46.990811 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:46.990812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:46.990813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:46.990819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:46.992293 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:46.992697 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:46.992699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:46.992699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:46.992751 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:46.992937 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:46.992938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:46.992938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:46.993050 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:46.993117 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:46.993314 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 13:41:46.993388 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:41:46.993813 14687 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:47.002512 14687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5ca81dfd4be4c12acf0dcc3b790944f: Generated new TSK 0
I20260924 13:41:47.002694 14687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:47.004526 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 14605
2026-09-24T13:41:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:47 dist-test-slave-m5z1 krb5kdc[14704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:47 dist-test-slave-m5z1 krb5kdc[14704](info): set up 2 sockets
Sep 24 13:41:47 dist-test-slave-m5z1 krb5kdc[14704](info): commencing operation
krb5kdc: starting...
W20260924 13:41:48.041953 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:48 dist-test-slave-m5z1 krb5kdc[14704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:48.158350 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:48.237072 14717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:48.237310 14717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:48.237349 14717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:48.238557 14717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:48.238626 14717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:48.238647 14717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:48.238667 14717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:48.238698 14717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:48.240140 14717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:48.240410 14717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:48.240638 14717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:48.245368 14723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:48.245538 14722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:48.245657 14717 server_base.cc:1034] running on GCE node
W20260924 13:41:48.245383 14725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:48.245965 14717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:48.246209 14717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:48.247344 14717 hybrid_clock.cc:648] HybridClock initialized: now 1790257308247324 us; error 34 us; skew 500 ppm
Sep 24 13:41:48 dist-test-slave-m5z1 krb5kdc[14704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257308, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:48.249729 14717 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:48.250188 14717 webserver.cc:460] Webserver started at http://127.0.0.1:33155/ using document root <none> and password file <none>
I20260924 13:41:48.250422 14717 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:48.250487 14717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:48.250620 14717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:48.251609 14717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "a7978154bc6a45a7808ac6aebc09ff0b"
format_stamp: "Formatted at 2026-09-24 13:41:48 on dist-test-slave-m5z1"
server_key: "8c392881a2fabba8219c962b265d91ff"
server_key_iv: "042c2b70f35fd12eaee1fb82e6205adb"
server_key_version: "encryptionkey@0"
I20260924 13:41:48.251963 14717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "a7978154bc6a45a7808ac6aebc09ff0b"
format_stamp: "Formatted at 2026-09-24 13:41:48 on dist-test-slave-m5z1"
server_key: "8c392881a2fabba8219c962b265d91ff"
server_key_iv: "042c2b70f35fd12eaee1fb82e6205adb"
server_key_version: "encryptionkey@0"
I20260924 13:41:48.253229 14717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:48.254014 14732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:48.254179 14717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:48.254274 14717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "a7978154bc6a45a7808ac6aebc09ff0b"
format_stamp: "Formatted at 2026-09-24 13:41:48 on dist-test-slave-m5z1"
server_key: "8c392881a2fabba8219c962b265d91ff"
server_key_iv: "042c2b70f35fd12eaee1fb82e6205adb"
server_key_version: "encryptionkey@0"
I20260924 13:41:48.254352 14717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:48.262392 14717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:48.264082 14717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:48.264237 14717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:48.268203 14717 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39835
I20260924 13:41:48.268239 14784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39835 every 8 connection(s)
I20260924 13:41:48.268575 14717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:48.269379 14785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:48.271567 14785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b: Bootstrap starting.
I20260924 13:41:48.272080 14785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:48.272311 14785 log.cc:824] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:48.272615 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 14717
I20260924 13:41:48.272728 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:48.272924 26561 external_mini_cluster.cc:1428] Setting key a61302ab88d091820bb6bc010c77bbd5
I20260924 13:41:48.273057 14785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b: No bootstrap required, opened a new log
I20260924 13:41:48.275972 14785 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7978154bc6a45a7808ac6aebc09ff0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39835 } }
I20260924 13:41:48.276136 14785 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:48.276165 14785 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7978154bc6a45a7808ac6aebc09ff0b, State: Initialized, Role: FOLLOWER
I20260924 13:41:48.276278 14785 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7978154bc6a45a7808ac6aebc09ff0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39835 } }
I20260924 13:41:48.276368 14785 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:48.276401 14785 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:48.276448 14785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:48.276983 14785 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7978154bc6a45a7808ac6aebc09ff0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39835 } }
I20260924 13:41:48.277070 14785 leader_election.cc:302] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7978154bc6a45a7808ac6aebc09ff0b; no voters:
I20260924 13:41:48.277241 14785 leader_election.cc:288] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:48.277263 14790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:48.277382 14790 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [term 1 LEADER]: Becoming Leader. State: Replica: a7978154bc6a45a7808ac6aebc09ff0b, State: Running, Role: LEADER
I20260924 13:41:48.277508 14785 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:48.277514 14790 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7978154bc6a45a7808ac6aebc09ff0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39835 } }
I20260924 13:41:48.277980 14791 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7978154bc6a45a7808ac6aebc09ff0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7978154bc6a45a7808ac6aebc09ff0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39835 } } }
I20260924 13:41:48.277995 14792 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7978154bc6a45a7808ac6aebc09ff0b. Latest consensus state: current_term: 1 leader_uuid: "a7978154bc6a45a7808ac6aebc09ff0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7978154bc6a45a7808ac6aebc09ff0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39835 } } }
I20260924 13:41:48.278221 14792 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:48.278138 14791 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:48.278504 14799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:48.279100 14799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:48.279608 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:48.273434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44415 (local address 127.0.0.1:39835)
0924 13:41:48.273648 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:48.273657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:48.273694 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:41:48.274464 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:48.274474 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:48.274536 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:48.274690 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:48.274871 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:48.276266 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:48.277124 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:48.277745 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:48.278204 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:48.278665 (+ 461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:48.278670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:48.278670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:48.278880 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:48.278921 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:48.278936 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:48.279229 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:48.279321 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 13:41:48.279383 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260924 13:41:48.280742 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:48.280947 14799 catalog_manager.cc:1347] Generated new cluster ID: e36df5fe51d347c2a7e1f15a96037fa7
I20260924 13:41:48.280998 14799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:48.288352 14789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:48.282078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44369 (local address 127.0.0.1:39835)
0924 13:41:48.282116 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:48.282117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:48.285737 (+ 3620us) server_negotiation.cc:407] Connection header received
0924 13:41:48.285835 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:48.285836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:48.285859 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:48.285889 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:48.286072 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:48.286518 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:48.287220 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:48.287414 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:48.287494 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:48.288126 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:48.288127 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:48.288127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:48.288131 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:48.288144 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:48.288148 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:48.288195 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:48.288248 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 13:41:48.288268 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:41:48.291949 14799 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:48.292433 14799 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:48.298168 14799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7978154bc6a45a7808ac6aebc09ff0b: Generated new TSK 0
I20260924 13:41:48.298339 14799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:48.300452 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 14717
2026-09-24T13:41:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:48 dist-test-slave-m5z1 krb5kdc[14816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:48 dist-test-slave-m5z1 krb5kdc[14816](info): set up 2 sockets
Sep 24 13:41:48 dist-test-slave-m5z1 krb5kdc[14816](info): commencing operation
krb5kdc: starting...
W20260924 13:41:49.322027 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:49 dist-test-slave-m5z1 krb5kdc[14816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:49.439822 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:49.521068 14829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:49.521298 14829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:49.521337 14829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:49.522518 14829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:49.522589 14829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:49.522614 14829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:49.522635 14829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:49.522665 14829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:49.524163 14829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:49.524441 14829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:49.524662 14829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:49.529461 14835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:49.529497 14829 server_base.cc:1034] running on GCE node
W20260924 13:41:49.529452 14837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:49.529452 14834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:49.529915 14829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:49.530236 14829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:49.531416 14829 hybrid_clock.cc:648] HybridClock initialized: now 1790257309531408 us; error 35 us; skew 500 ppm
Sep 24 13:41:49 dist-test-slave-m5z1 krb5kdc[14816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:49.538347 14829 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:49.538864 14829 webserver.cc:460] Webserver started at http://127.0.0.1:44003/ using document root <none> and password file <none>
I20260924 13:41:49.539152 14829 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:49.539229 14829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:49.539386 14829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:49.540500 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "68084e66b3d045a5ae283f6ef9b24dce"
format_stamp: "Formatted at 2026-09-24 13:41:49 on dist-test-slave-m5z1"
server_key: "dbff47e33f6925616d78480fcf834fb0"
server_key_iv: "5b54b5c631fa790da6e93054b356bb68"
server_key_version: "encryptionkey@0"
I20260924 13:41:49.540880 14829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "68084e66b3d045a5ae283f6ef9b24dce"
format_stamp: "Formatted at 2026-09-24 13:41:49 on dist-test-slave-m5z1"
server_key: "dbff47e33f6925616d78480fcf834fb0"
server_key_iv: "5b54b5c631fa790da6e93054b356bb68"
server_key_version: "encryptionkey@0"
I20260924 13:41:49.542357 14829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:41:49.543136 14844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:49.543308 14829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:49.543421 14829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "68084e66b3d045a5ae283f6ef9b24dce"
format_stamp: "Formatted at 2026-09-24 13:41:49 on dist-test-slave-m5z1"
server_key: "dbff47e33f6925616d78480fcf834fb0"
server_key_iv: "5b54b5c631fa790da6e93054b356bb68"
server_key_version: "encryptionkey@0"
I20260924 13:41:49.543501 14829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:49.547186 14829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:49.549093 14829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:49.549261 14829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:49.553854 14829 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46781
I20260924 13:41:49.553925 14896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46781 every 8 connection(s)
I20260924 13:41:49.554391 14829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:49.555599 14897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:49.557875 14897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce: Bootstrap starting.
I20260924 13:41:49.558436 14897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:49.558712 14897 log.cc:824] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:49.559391 14897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce: No bootstrap required, opened a new log
I20260924 13:41:49.560901 14897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68084e66b3d045a5ae283f6ef9b24dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46781 } }
I20260924 13:41:49.561029 14897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:49.561051 14897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68084e66b3d045a5ae283f6ef9b24dce, State: Initialized, Role: FOLLOWER
I20260924 13:41:49.561234 14897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68084e66b3d045a5ae283f6ef9b24dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46781 } }
I20260924 13:41:49.561296 14897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:49.561316 14897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:49.561379 14897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:49.561961 14897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68084e66b3d045a5ae283f6ef9b24dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46781 } }
I20260924 13:41:49.562070 14897 leader_election.cc:302] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68084e66b3d045a5ae283f6ef9b24dce; no voters:
I20260924 13:41:49.562224 14897 leader_election.cc:288] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:49.562299 14900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:49.562459 14897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:49.562481 14900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [term 1 LEADER]: Becoming Leader. State: Replica: 68084e66b3d045a5ae283f6ef9b24dce, State: Running, Role: LEADER
I20260924 13:41:49.562701 14900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68084e66b3d045a5ae283f6ef9b24dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46781 } }
I20260924 13:41:49.563352 14902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68084e66b3d045a5ae283f6ef9b24dce. Latest consensus state: current_term: 1 leader_uuid: "68084e66b3d045a5ae283f6ef9b24dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68084e66b3d045a5ae283f6ef9b24dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46781 } } }
I20260924 13:41:49.563438 14902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:49.563570 14901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68084e66b3d045a5ae283f6ef9b24dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68084e66b3d045a5ae283f6ef9b24dce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46781 } } }
I20260924 13:41:49.563657 14901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:49.564152 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 14829
I20260924 13:41:49.564375 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
W20260924 13:41:49.564405 14915 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:41:49.564450 14915 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:41:49.564509 14910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:49.564671 26561 external_mini_cluster.cc:1428] Setting key f1d56dc915430f4b47526225e5a9659a
I20260924 13:41:49.564679 14910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:49.566201 14910 catalog_manager.cc:1347] Generated new cluster ID: 79d5032337c74d4cbb1df95e43045bf1
I20260924 13:41:49.566252 14910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:49.567271 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:49.565187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43865 (local address 127.0.0.1:46781)
0924 13:41:49.565349 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:49.565358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:49.565765 (+ 407us) server_negotiation.cc:407] Connection header received
0924 13:41:49.565989 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:49.565995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:49.566019 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:49.566167 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:49.566836 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:49.566840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:49.566840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:49.566845 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:49.566895 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:49.566907 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:49.566994 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:49.567074 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 13:41:49.567118 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260924 13:41:49.568127 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:49.570439 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:49.569351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57281 (local address 127.0.0.1:46781)
0924 13:41:49.569383 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:49.569385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:49.569448 (+ 63us) server_negotiation.cc:407] Connection header received
0924 13:41:49.569508 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:49.569509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:49.569512 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:49.569606 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:49.570233 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:49.570234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:49.570234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:49.570239 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:49.570253 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:49.570258 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:49.570316 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:49.570371 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 13:41:49.570384 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 13:41:49.575436 14910 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:49.575584 14910 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:49.579211 14910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68084e66b3d045a5ae283f6ef9b24dce: Generated new TSK 0
I20260924 13:41:49.579372 14910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:49.583086 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 14829
2026-09-24T13:41:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:49 dist-test-slave-m5z1 krb5kdc[14928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:49 dist-test-slave-m5z1 krb5kdc[14928](info): set up 2 sockets
Sep 24 13:41:49 dist-test-slave-m5z1 krb5kdc[14928](info): commencing operation
krb5kdc: starting...
W20260924 13:41:50.617885 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:50 dist-test-slave-m5z1 krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:50.740164 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41251
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:50.820349 14941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:50.820542 14941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:50.820565 14941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:50.821762 14941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:50.821803 14941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:50.821817 14941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:50.821830 14941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:50.821842 14941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:50.823208 14941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:50.823383 14941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:50.823539 14941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:50.828467 14946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:50.828497 14949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:50.828446 14947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:50.828650 14941 server_base.cc:1034] running on GCE node
I20260924 13:41:50.828867 14941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:50.829162 14941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:50.830322 14941 hybrid_clock.cc:648] HybridClock initialized: now 1790257310830311 us; error 44 us; skew 500 ppm
Sep 24 13:41:50 dist-test-slave-m5z1 krb5kdc[14928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:50.832607 14941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:50.833077 14941 webserver.cc:460] Webserver started at http://127.0.0.1:39905/ using document root <none> and password file <none>
I20260924 13:41:50.833298 14941 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:50.833364 14941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:50.833498 14941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:50.834587 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "a1b2070efe214b64ae112cd8a43290a6"
format_stamp: "Formatted at 2026-09-24 13:41:50 on dist-test-slave-m5z1"
server_key: "fd4eba3929cfd6a99b03c1d49d1b51d0"
server_key_iv: "f532dabc659a3a89fad4dde4cfec8dea"
server_key_version: "encryptionkey@0"
I20260924 13:41:50.834949 14941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "a1b2070efe214b64ae112cd8a43290a6"
format_stamp: "Formatted at 2026-09-24 13:41:50 on dist-test-slave-m5z1"
server_key: "fd4eba3929cfd6a99b03c1d49d1b51d0"
server_key_iv: "f532dabc659a3a89fad4dde4cfec8dea"
server_key_version: "encryptionkey@0"
I20260924 13:41:50.836218 14941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:41:50.836979 14956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:50.837162 14941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:50.837258 14941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "a1b2070efe214b64ae112cd8a43290a6"
format_stamp: "Formatted at 2026-09-24 13:41:50 on dist-test-slave-m5z1"
server_key: "fd4eba3929cfd6a99b03c1d49d1b51d0"
server_key_iv: "f532dabc659a3a89fad4dde4cfec8dea"
server_key_version: "encryptionkey@0"
I20260924 13:41:50.837339 14941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:50.881603 14941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:50.884325 14941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:50.884438 14941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:50.888356 14941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41251
I20260924 13:41:50.888405 15008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41251 every 8 connection(s)
I20260924 13:41:50.888698 14941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:50.889519 15009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:50.891798 15009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6: Bootstrap starting.
I20260924 13:41:50.892333 15009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:50.892561 15009 log.cc:824] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:50.893230 15009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6: No bootstrap required, opened a new log
I20260924 13:41:50.894811 15009 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b2070efe214b64ae112cd8a43290a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260924 13:41:50.895035 15009 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:50.895032 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 14941
I20260924 13:41:50.895087 15009 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b2070efe214b64ae112cd8a43290a6, State: Initialized, Role: FOLLOWER
I20260924 13:41:50.895152 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:50.895241 15009 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b2070efe214b64ae112cd8a43290a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260924 13:41:50.895321 15009 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:50.895357 15009 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:50.895373 26561 external_mini_cluster.cc:1428] Setting key d764901303e5fc83b129ebfeb7317bfa
I20260924 13:41:50.895408 15009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:50.896011 15009 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b2070efe214b64ae112cd8a43290a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260924 13:41:50.896136 15009 leader_election.cc:302] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b2070efe214b64ae112cd8a43290a6; no voters:
I20260924 13:41:50.896292 15009 leader_election.cc:288] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:50.896320 15014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:50.896497 15014 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [term 1 LEADER]: Becoming Leader. State: Replica: a1b2070efe214b64ae112cd8a43290a6, State: Running, Role: LEADER
I20260924 13:41:50.896548 15009 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:50.896651 15014 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b2070efe214b64ae112cd8a43290a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } }
I20260924 13:41:50.897260 15014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b2070efe214b64ae112cd8a43290a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b2070efe214b64ae112cd8a43290a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } } }
I20260924 13:41:50.897390 15014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:50.897598 15014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b2070efe214b64ae112cd8a43290a6. Latest consensus state: current_term: 1 leader_uuid: "a1b2070efe214b64ae112cd8a43290a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b2070efe214b64ae112cd8a43290a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41251 } } }
I20260924 13:41:50.897687 15014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:50.898072 15023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:50.899520 15023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:50.901331 15023 catalog_manager.cc:1347] Generated new cluster ID: ae1ccbc137f74b8f832f7a8c1dab182e
I20260924 13:41:50.901394 15023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:41:50 dist-test-slave-m5z1 krb5kdc[14928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:41:50.909946 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:50.895874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:40955 (local address 127.0.0.1:41251)
0924 13:41:50.896048 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:50.896058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:50.896110 (+ 52us) server_negotiation.cc:407] Connection header received
0924 13:41:50.896250 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:50.896258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:50.896321 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:50.896507 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:50.897353 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:50.898706 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:50.899889 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:50.900509 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:50.900629 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:50.903742 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:50.903747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:50.903747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:50.903760 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:50.908096 (+ 4336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:50.908596 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:50.908599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:50.908599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:50.908721 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:50.908971 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:50.908972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:50.908972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:50.909299 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:50.909475 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:50.909609 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 13:41:50.909719 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260924 13:41:50.911065 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:50.920223 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:50.912624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:33591 (local address 127.0.0.1:41251)
0924 13:41:50.912666 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:50.912667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:50.912826 (+ 159us) server_negotiation.cc:407] Connection header received
0924 13:41:50.912864 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:50.912865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:50.912876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:50.912940 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:50.913593 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:50.914392 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:50.915476 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:50.915830 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:50.915902 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:50.917298 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:50.917299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:50.917299 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:50.917307 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:50.918947 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:50.919385 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:50.919386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:50.919387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:50.919447 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:50.919674 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:50.919675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:50.919675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:50.919805 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:50.919910 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:50.920063 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 13:41:50.920130 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":23}
I20260924 13:41:50.931229 15023 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:50.931975 15023 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:50.936518 15023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b2070efe214b64ae112cd8a43290a6: Generated new TSK 0
I20260924 13:41:50.936700 15023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:50.941949 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 14941
W20260924 13:41:50.947098 14926 connection.cc:535] client connection to 127.0.0.1:41251 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41251): Connection reset by peer (error 104)
2026-09-24T13:41:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:51 dist-test-slave-m5z1 krb5kdc[15040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:51 dist-test-slave-m5z1 krb5kdc[15040](info): set up 2 sockets
Sep 24 13:41:51 dist-test-slave-m5z1 krb5kdc[15040](info): commencing operation
krb5kdc: starting...
W20260924 13:41:51.993834 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:52 dist-test-slave-m5z1 krb5kdc[15040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:52.111726 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:52.190658 15053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:52.190851 15053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:52.190874 15053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:52.191923 15053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:52.191958 15053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:52.191972 15053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:52.191985 15053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:52.191998 15053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:52.193343 15053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:52.193521 15053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:52.193765 15053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:52.198375 15061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:52.198393 15058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:52.198421 15059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:52.198678 15053 server_base.cc:1034] running on GCE node
I20260924 13:41:52.198876 15053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:52.199108 15053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:52.200248 15053 hybrid_clock.cc:648] HybridClock initialized: now 1790257312200224 us; error 37 us; skew 500 ppm
Sep 24 13:41:52 dist-test-slave-m5z1 krb5kdc[15040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257312, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:52.202631 15053 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:52.203048 15053 webserver.cc:460] Webserver started at http://127.0.0.1:33307/ using document root <none> and password file <none>
I20260924 13:41:52.203262 15053 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:52.203328 15053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:52.203476 15053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:52.204485 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "36fc380cd8f0414d9180837fc86b7250"
format_stamp: "Formatted at 2026-09-24 13:41:52 on dist-test-slave-m5z1"
server_key: "76b9a46b3bc0f63a1bfe73966f2dc365"
server_key_iv: "bcb222fb6c6b30e67c74effc3b330e7a"
server_key_version: "encryptionkey@0"
I20260924 13:41:52.204835 15053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "36fc380cd8f0414d9180837fc86b7250"
format_stamp: "Formatted at 2026-09-24 13:41:52 on dist-test-slave-m5z1"
server_key: "76b9a46b3bc0f63a1bfe73966f2dc365"
server_key_iv: "bcb222fb6c6b30e67c74effc3b330e7a"
server_key_version: "encryptionkey@0"
I20260924 13:41:52.206214 15053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 13:41:52.206887 15068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:52.207053 15053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:52.207155 15053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "36fc380cd8f0414d9180837fc86b7250"
format_stamp: "Formatted at 2026-09-24 13:41:52 on dist-test-slave-m5z1"
server_key: "76b9a46b3bc0f63a1bfe73966f2dc365"
server_key_iv: "bcb222fb6c6b30e67c74effc3b330e7a"
server_key_version: "encryptionkey@0"
I20260924 13:41:52.207227 15053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:52.216322 15053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:52.218065 15053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:52.218211 15053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:52.221895 15053 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39175
I20260924 13:41:52.221923 15120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39175 every 8 connection(s)
I20260924 13:41:52.222273 15053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:52.223043 15121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:52.225533 15121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250: Bootstrap starting.
I20260924 13:41:52.226151 15121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:52.226244 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15053
I20260924 13:41:52.226377 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:52.226493 15121 log.cc:824] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:52.226590 26561 external_mini_cluster.cc:1428] Setting key 5c938e4111eadc1031d459bc4507e94f
I20260924 13:41:52.227303 15121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250: No bootstrap required, opened a new log
I20260924 13:41:52.229262 15121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc380cd8f0414d9180837fc86b7250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39175 } }
I20260924 13:41:52.229388 15121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:52.229411 15121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fc380cd8f0414d9180837fc86b7250, State: Initialized, Role: FOLLOWER
I20260924 13:41:52.229543 15121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc380cd8f0414d9180837fc86b7250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39175 } }
I20260924 13:41:52.229662 15121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:52.229717 15121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:52.229768 15121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:52.230273 15121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc380cd8f0414d9180837fc86b7250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39175 } }
I20260924 13:41:52.230389 15121 leader_election.cc:302] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36fc380cd8f0414d9180837fc86b7250; no voters:
I20260924 13:41:52.230587 15121 leader_election.cc:288] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:52.230607 15126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:52.230906 15121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:52.230800 15126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [term 1 LEADER]: Becoming Leader. State: Replica: 36fc380cd8f0414d9180837fc86b7250, State: Running, Role: LEADER
I20260924 13:41:52.231261 15126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc380cd8f0414d9180837fc86b7250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39175 } }
I20260924 13:41:52.231709 15126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36fc380cd8f0414d9180837fc86b7250. Latest consensus state: current_term: 1 leader_uuid: "36fc380cd8f0414d9180837fc86b7250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc380cd8f0414d9180837fc86b7250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39175 } } }
I20260924 13:41:52.231993 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:52.227051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:51173 (local address 127.0.0.1:39175)
0924 13:41:52.227206 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:52.227216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:52.227251 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:41:52.227721 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:52.227727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:52.227775 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:52.227914 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:52.228120 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:52.228992 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:52.230015 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:52.230498 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:52.230603 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:52.231311 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:52.231315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:52.231315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:52.231531 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:52.231571 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:52.231588 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:52.231671 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:52.231752 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 13:41:52.231778 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260924 13:41:52.232605 15126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:52.232033 15127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36fc380cd8f0414d9180837fc86b7250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36fc380cd8f0414d9180837fc86b7250" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39175 } } }
I20260924 13:41:52.232786 15127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250 [sys.catalog]: This master's current role is: LEADER
W20260924 13:41:52.232851 15140 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:41:52.232954 15140 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:41:52.233026 15143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:52.233194 15143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:52.233189 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:52.234640 15143 catalog_manager.cc:1347] Generated new cluster ID: 7e75bfc0fb0a42369b4bf186b2dcdee2
I20260924 13:41:52.234690 15143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:52.237535 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:52.234791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:32793 (local address 127.0.0.1:39175)
0924 13:41:52.234825 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:52.234827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:52.234899 (+ 72us) server_negotiation.cc:407] Connection header received
0924 13:41:52.234975 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:52.234976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:52.234986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:52.235039 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:52.235160 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:52.235625 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:52.236345 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:52.236570 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:52.236642 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:52.237275 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:52.237277 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:52.237277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:52.237281 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:52.237294 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:52.237298 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:52.237355 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:52.237427 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 13:41:52.237443 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 13:41:52.268797 15143 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:52.269354 15143 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:52.273116 15143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36fc380cd8f0414d9180837fc86b7250: Generated new TSK 0
I20260924 13:41:52.273316 15143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:52.281308 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15053
W20260924 13:41:52.285498 15038 connection.cc:535] client connection to 127.0.0.1:39175 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39175): Connection reset by peer (error 104)
2026-09-24T13:41:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:52 dist-test-slave-m5z1 krb5kdc[15152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:52 dist-test-slave-m5z1 krb5kdc[15152](info): set up 2 sockets
Sep 24 13:41:52 dist-test-slave-m5z1 krb5kdc[15152](info): commencing operation
krb5kdc: starting...
W20260924 13:41:53.313935 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:53 dist-test-slave-m5z1 krb5kdc[15152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:53.431663 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:53.510865 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:53.511108 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:53.511149 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:53.512291 15165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:53.512353 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:53.512379 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:53.512411 15165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:53.512437 15165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:53.513922 15165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35453
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:53.514197 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:53.514415 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:53.519394 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:53.519383 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:53.519436 15165 server_base.cc:1034] running on GCE node
W20260924 13:41:53.519387 15173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:53.519819 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:53.520119 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:53.521266 15165 hybrid_clock.cc:648] HybridClock initialized: now 1790257313521256 us; error 31 us; skew 500 ppm
Sep 24 13:41:53 dist-test-slave-m5z1 krb5kdc[15152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:53.523825 15165 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:53.524274 15165 webserver.cc:460] Webserver started at http://127.0.0.1:46069/ using document root <none> and password file <none>
I20260924 13:41:53.524497 15165 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:53.524567 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:53.524713 15165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:53.525825 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "92370f67d8d84a668f8ead308ae82ac9"
format_stamp: "Formatted at 2026-09-24 13:41:53 on dist-test-slave-m5z1"
server_key: "a02817cc06914335071ec31c1635c29b"
server_key_iv: "b861243e280f35b533d898a5eca654c0"
server_key_version: "encryptionkey@0"
I20260924 13:41:53.526180 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "92370f67d8d84a668f8ead308ae82ac9"
format_stamp: "Formatted at 2026-09-24 13:41:53 on dist-test-slave-m5z1"
server_key: "a02817cc06914335071ec31c1635c29b"
server_key_iv: "b861243e280f35b533d898a5eca654c0"
server_key_version: "encryptionkey@0"
I20260924 13:41:53.527474 15165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:53.528229 15180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:53.528421 15165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:53.528527 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "92370f67d8d84a668f8ead308ae82ac9"
format_stamp: "Formatted at 2026-09-24 13:41:53 on dist-test-slave-m5z1"
server_key: "a02817cc06914335071ec31c1635c29b"
server_key_iv: "b861243e280f35b533d898a5eca654c0"
server_key_version: "encryptionkey@0"
I20260924 13:41:53.528601 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:53.531881 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:53.534106 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:53.534256 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:53.537988 15165 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35453
I20260924 13:41:53.538023 15232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35453 every 8 connection(s)
I20260924 13:41:53.538345 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:53.539268 15233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:53.541402 15233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9: Bootstrap starting.
I20260924 13:41:53.542011 15233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:53.542291 15233 log.cc:824] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:53.542932 15233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9: No bootstrap required, opened a new log
I20260924 13:41:53.544462 15233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92370f67d8d84a668f8ead308ae82ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260924 13:41:53.544595 15233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:53.544616 15233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92370f67d8d84a668f8ead308ae82ac9, State: Initialized, Role: FOLLOWER
I20260924 13:41:53.544764 15233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92370f67d8d84a668f8ead308ae82ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260924 13:41:53.544845 15233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:53.544871 15233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:53.544936 15233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:53.545477 15233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92370f67d8d84a668f8ead308ae82ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260924 13:41:53.545612 15233 leader_election.cc:302] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92370f67d8d84a668f8ead308ae82ac9; no voters:
I20260924 13:41:53.545773 15233 leader_election.cc:288] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:53.545804 15236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:53.545939 15236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [term 1 LEADER]: Becoming Leader. State: Replica: 92370f67d8d84a668f8ead308ae82ac9, State: Running, Role: LEADER
I20260924 13:41:53.546031 15233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:53.546097 15236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92370f67d8d84a668f8ead308ae82ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260924 13:41:53.546226 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15165
I20260924 13:41:53.546358 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:53.546586 26561 external_mini_cluster.cc:1428] Setting key 8a023de62cbb691f2d34e9363c1fe8b1
I20260924 13:41:53.547242 15237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92370f67d8d84a668f8ead308ae82ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92370f67d8d84a668f8ead308ae82ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } } }
I20260924 13:41:53.547406 15237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:53.547235 15238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92370f67d8d84a668f8ead308ae82ac9. Latest consensus state: current_term: 1 leader_uuid: "92370f67d8d84a668f8ead308ae82ac9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92370f67d8d84a668f8ead308ae82ac9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } } }
I20260924 13:41:53.547576 15238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9 [sys.catalog]: This master's current role is: LEADER
W20260924 13:41:53.547932 15252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:41:53.547976 15252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:41:53.548038 15253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:53.548215 15253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:53.549408 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:53.547039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:33363 (local address 127.0.0.1:35453)
0924 13:41:53.548010 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:53.548019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:53.548052 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:41:53.548165 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:53.548172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:53.548202 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:53.548374 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:53.549016 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:53.549019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:53.549019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:53.549023 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:53.549056 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:53.549073 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:53.549146 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:53.549217 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 13:41:53.549259 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":85,"threads_started":1}
I20260924 13:41:53.550387 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:53.551754 15253 catalog_manager.cc:1347] Generated new cluster ID: 1c423a98543b4b62846c39783baf6fa1
I20260924 13:41:53.551808 15253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:53.552846 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:53.551722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:50133 (local address 127.0.0.1:35453)
0924 13:41:53.551757 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:53.551759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:53.551865 (+ 106us) server_negotiation.cc:407] Connection header received
0924 13:41:53.551950 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:53.551951 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:53.551956 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:53.552011 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:53.552604 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:53.552605 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:53.552606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:53.552610 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:53.552624 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:53.552628 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:53.552683 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:53.552739 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 13:41:53.552754 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:41:53.570334 15253 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:53.570536 15253 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:53.575553 15253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92370f67d8d84a668f8ead308ae82ac9: Generated new TSK 0
I20260924 13:41:53.575732 15253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:53.584625 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15165
2026-09-24T13:41:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:53 dist-test-slave-m5z1 krb5kdc[15264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:53 dist-test-slave-m5z1 krb5kdc[15264](info): set up 2 sockets
Sep 24 13:41:53 dist-test-slave-m5z1 krb5kdc[15264](info): commencing operation
krb5kdc: starting...
W20260924 13:41:54.613837 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:54 dist-test-slave-m5z1 krb5kdc[15264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:54.730367 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:54.809321 15277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:54.809517 15277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:54.809540 15277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:54.810781 15277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:54.810825 15277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:54.810840 15277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:54.810853 15277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:54.810905 15277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:54.812326 15277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:54.812566 15277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:54.812777 15277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:54.817513 15283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:54.817678 15277 server_base.cc:1034] running on GCE node
W20260924 13:41:54.817503 15285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:54.817503 15282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:54.818058 15277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:54.818369 15277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:54.819535 15277 hybrid_clock.cc:648] HybridClock initialized: now 1790257314819522 us; error 24 us; skew 500 ppm
Sep 24 13:41:54 dist-test-slave-m5z1 krb5kdc[15264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:54.826632 15277 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:54.827076 15277 webserver.cc:460] Webserver started at http://127.0.0.1:34979/ using document root <none> and password file <none>
I20260924 13:41:54.827310 15277 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:54.827378 15277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:54.827526 15277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:54.828568 15277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "ba8d67ea2ec0427892ff17b3ed5bea93"
format_stamp: "Formatted at 2026-09-24 13:41:54 on dist-test-slave-m5z1"
server_key: "ad2579de500260d3133779237c4b542e"
server_key_iv: "0cfacf6d44fe5be52b85c68eb5d5d6b9"
server_key_version: "encryptionkey@0"
I20260924 13:41:54.828928 15277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "ba8d67ea2ec0427892ff17b3ed5bea93"
format_stamp: "Formatted at 2026-09-24 13:41:54 on dist-test-slave-m5z1"
server_key: "ad2579de500260d3133779237c4b542e"
server_key_iv: "0cfacf6d44fe5be52b85c68eb5d5d6b9"
server_key_version: "encryptionkey@0"
I20260924 13:41:54.830279 15277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:41:54.830960 15292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:54.831125 15277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:41:54.831228 15277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "ba8d67ea2ec0427892ff17b3ed5bea93"
format_stamp: "Formatted at 2026-09-24 13:41:54 on dist-test-slave-m5z1"
server_key: "ad2579de500260d3133779237c4b542e"
server_key_iv: "0cfacf6d44fe5be52b85c68eb5d5d6b9"
server_key_version: "encryptionkey@0"
I20260924 13:41:54.831305 15277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:54.844595 15277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:54.846378 15277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:54.846539 15277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:54.850248 15277 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42135
I20260924 13:41:54.850294 15344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42135 every 8 connection(s)
I20260924 13:41:54.850628 15277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:54.851428 15345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:54.853544 15345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93: Bootstrap starting.
I20260924 13:41:54.854171 15345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:54.854503 15345 log.cc:824] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:54.854547 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15277
I20260924 13:41:54.854653 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:54.854828 26561 external_mini_cluster.cc:1428] Setting key 870f53f47a284af9391d530956617e04
I20260924 13:41:54.855301 15345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93: No bootstrap required, opened a new log
I20260924 13:41:54.856920 15345 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8d67ea2ec0427892ff17b3ed5bea93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42135 } }
I20260924 13:41:54.857081 15345 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:54.857128 15345 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8d67ea2ec0427892ff17b3ed5bea93, State: Initialized, Role: FOLLOWER
I20260924 13:41:54.857281 15345 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8d67ea2ec0427892ff17b3ed5bea93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42135 } }
I20260924 13:41:54.857374 15345 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:54.857419 15345 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:54.857471 15345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:54.858352 15345 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8d67ea2ec0427892ff17b3ed5bea93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42135 } }
I20260924 13:41:54.858471 15345 leader_election.cc:302] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba8d67ea2ec0427892ff17b3ed5bea93; no voters:
I20260924 13:41:54.858660 15345 leader_election.cc:288] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:54.858681 15350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:54.858790 15350 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [term 1 LEADER]: Becoming Leader. State: Replica: ba8d67ea2ec0427892ff17b3ed5bea93, State: Running, Role: LEADER
I20260924 13:41:54.858872 15350 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8d67ea2ec0427892ff17b3ed5bea93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42135 } }
I20260924 13:41:54.858919 15345 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:54.859216 15352 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba8d67ea2ec0427892ff17b3ed5bea93. Latest consensus state: current_term: 1 leader_uuid: "ba8d67ea2ec0427892ff17b3ed5bea93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8d67ea2ec0427892ff17b3ed5bea93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42135 } } }
I20260924 13:41:54.859300 15352 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:54.860404 15351 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba8d67ea2ec0427892ff17b3ed5bea93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba8d67ea2ec0427892ff17b3ed5bea93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42135 } } }
I20260924 13:41:54.860482 15351 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93 [sys.catalog]: This master's current role is: LEADER
W20260924 13:41:54.860792 15366 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:41:54.860838 15366 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:41:54.860898 15355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:54.861029 15355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:54.862465 15355 catalog_manager.cc:1347] Generated new cluster ID: 75cb86b4e2c044f38c1fec91c69d0a8c
I20260924 13:41:54.862519 15355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:41:54 dist-test-slave-m5z1 krb5kdc[15264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 13:41:54.866616 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:54.855375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:44645 (local address 127.0.0.1:42135)
0924 13:41:54.855568 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:54.855579 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:54.855616 (+ 37us) server_negotiation.cc:407] Connection header received
0924 13:41:54.856916 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:54.856923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:54.856978 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:54.857151 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:54.857921 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:54.859069 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:54.860148 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:54.860731 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:54.860876 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:54.863415 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:54.863419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:54.863419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:54.863431 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:54.865057 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:54.865456 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:54.865461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:54.865461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:54.865548 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:54.865839 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:54.865840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:54.865841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:54.866076 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:54.866227 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:54.866357 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 13:41:54.866457 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260924 13:41:54.867558 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:54.869309 15355 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:54.869864 15355 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:54.873919 15355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba8d67ea2ec0427892ff17b3ed5bea93: Generated new TSK 0
I20260924 13:41:54.874097 15355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:54.880476 15349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:54.873901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:41259 (local address 127.0.0.1:42135)
0924 13:41:54.873941 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:54.873943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:54.874178 (+ 235us) server_negotiation.cc:407] Connection header received
0924 13:41:54.874276 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:54.874276 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:54.874286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:54.874348 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:54.874976 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:54.875463 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:54.876502 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:54.876701 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:54.876750 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:54.878052 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:54.878053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:54.878054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:54.878059 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:54.879393 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:54.879804 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:54.879805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:54.879805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:54.879852 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:54.880073 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:54.880073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:54.880073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:54.880160 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:54.880232 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:54.880379 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 13:41:54.880422 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:41:54.882649 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15277
W20260924 13:41:54.886824 15262 connection.cc:535] client connection to 127.0.0.1:42135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42135): Connection reset by peer (error 104)
2026-09-24T13:41:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:54 dist-test-slave-m5z1 krb5kdc[15376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:54 dist-test-slave-m5z1 krb5kdc[15376](info): set up 2 sockets
Sep 24 13:41:54 dist-test-slave-m5z1 krb5kdc[15376](info): commencing operation
krb5kdc: starting...
W20260924 13:41:55.921937 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:56 dist-test-slave-m5z1 krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:56.040002 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:56.119812 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:56.120011 15389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:56.120034 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:56.121172 15389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:56.121213 15389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:56.121227 15389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:56.121240 15389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:56.121291 15389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:56.122766 15389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:56.123003 15389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:56.123219 15389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:56.127847 15397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:56.127854 15394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:56.127869 15395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:56.128211 15389 server_base.cc:1034] running on GCE node
I20260924 13:41:56.128429 15389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:56.128661 15389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:56.129792 15389 hybrid_clock.cc:648] HybridClock initialized: now 1790257316129781 us; error 34 us; skew 500 ppm
Sep 24 13:41:56 dist-test-slave-m5z1 krb5kdc[15376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:56.132247 15389 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:56.132655 15389 webserver.cc:460] Webserver started at http://127.0.0.1:32941/ using document root <none> and password file <none>
I20260924 13:41:56.132879 15389 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:56.132923 15389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:56.133055 15389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:56.134200 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "69591b9833bb4ae2b1036d470389c42a"
format_stamp: "Formatted at 2026-09-24 13:41:56 on dist-test-slave-m5z1"
server_key: "fe1e66e97a3043772068ee4073b576e6"
server_key_iv: "42ab2f270283811bc42e5a458a372e76"
server_key_version: "encryptionkey@0"
I20260924 13:41:56.134549 15389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "69591b9833bb4ae2b1036d470389c42a"
format_stamp: "Formatted at 2026-09-24 13:41:56 on dist-test-slave-m5z1"
server_key: "fe1e66e97a3043772068ee4073b576e6"
server_key_iv: "42ab2f270283811bc42e5a458a372e76"
server_key_version: "encryptionkey@0"
I20260924 13:41:56.135854 15389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:41:56.136651 15404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:56.136868 15389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:41:56.136966 15389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "69591b9833bb4ae2b1036d470389c42a"
format_stamp: "Formatted at 2026-09-24 13:41:56 on dist-test-slave-m5z1"
server_key: "fe1e66e97a3043772068ee4073b576e6"
server_key_iv: "42ab2f270283811bc42e5a458a372e76"
server_key_version: "encryptionkey@0"
I20260924 13:41:56.137038 15389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:56.153959 15389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:56.155781 15389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:56.155938 15389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:56.159814 15389 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40137
I20260924 13:41:56.159865 15456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40137 every 8 connection(s)
I20260924 13:41:56.160230 15389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:56.161041 15457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:56.163308 15457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a: Bootstrap starting.
I20260924 13:41:56.163866 15457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:56.164143 15457 log.cc:824] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:56.164441 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15389
I20260924 13:41:56.164597 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:56.164830 26561 external_mini_cluster.cc:1428] Setting key d4344cc3501a695d0a42c46a599f5ccc
I20260924 13:41:56.164861 15457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a: No bootstrap required, opened a new log
I20260924 13:41:56.167521 15457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591b9833bb4ae2b1036d470389c42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40137 } }
I20260924 13:41:56.167697 15457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:56.167737 15457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69591b9833bb4ae2b1036d470389c42a, State: Initialized, Role: FOLLOWER
I20260924 13:41:56.167860 15457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591b9833bb4ae2b1036d470389c42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40137 } }
I20260924 13:41:56.167936 15457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:56.167968 15457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:56.168004 15457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:56.168664 15457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591b9833bb4ae2b1036d470389c42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40137 } }
I20260924 13:41:56.168787 15457 leader_election.cc:302] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69591b9833bb4ae2b1036d470389c42a; no voters:
I20260924 13:41:56.168985 15457 leader_election.cc:288] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:56.169005 15462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:56.169122 15462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [term 1 LEADER]: Becoming Leader. State: Replica: 69591b9833bb4ae2b1036d470389c42a, State: Running, Role: LEADER
I20260924 13:41:56.169250 15457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:56.169257 15462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591b9833bb4ae2b1036d470389c42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40137 } }
I20260924 13:41:56.170048 15464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69591b9833bb4ae2b1036d470389c42a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591b9833bb4ae2b1036d470389c42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40137 } } }
I20260924 13:41:56.170468 15464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:56.170848 15475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:56.171110 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:56.165425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:35301 (local address 127.0.0.1:40137)
0924 13:41:56.165662 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:56.165670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:56.165697 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:41:56.166196 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:56.166203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:56.166253 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:56.166378 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:56.166698 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:56.167565 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:56.168707 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:56.169317 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:56.169414 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:56.170224 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:56.170229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:56.170229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:56.170431 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:56.170469 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:56.170483 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:56.170564 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:56.170820 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 13:41:56.170847 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":116,"threads_started":1}
I20260924 13:41:56.170374 15463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69591b9833bb4ae2b1036d470389c42a. Latest consensus state: current_term: 1 leader_uuid: "69591b9833bb4ae2b1036d470389c42a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69591b9833bb4ae2b1036d470389c42a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40137 } } }
I20260924 13:41:56.171233 15463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:56.171479 15475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:56.172047 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:56.172994 15475 catalog_manager.cc:1347] Generated new cluster ID: 6bc969a15bce4a17a6379850eac04974
I20260924 13:41:56.173041 15475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:56.176254 15461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:56.173616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:47943 (local address 127.0.0.1:40137)
0924 13:41:56.173653 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:56.173655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:56.173664 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:41:56.173746 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:56.173746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:56.173756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:56.173810 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:56.173933 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:56.174374 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:56.175104 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:56.175323 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:56.175398 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:56.176041 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:56.176042 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:56.176043 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:56.176047 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:56.176059 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:56.176063 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:56.176116 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:56.176175 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 13:41:56.176183 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:41:56.198309 15475 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:56.198858 15475 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:56.201651 15475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69591b9833bb4ae2b1036d470389c42a: Generated new TSK 0
I20260924 13:41:56.201835 15475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:56.210144 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15389
W20260924 13:41:56.214526 15374 connection.cc:535] client connection to 127.0.0.1:40137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40137): Connection reset by peer (error 104)
2026-09-24T13:41:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:56 dist-test-slave-m5z1 krb5kdc[15488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:56 dist-test-slave-m5z1 krb5kdc[15488](info): set up 2 sockets
Sep 24 13:41:56 dist-test-slave-m5z1 krb5kdc[15488](info): commencing operation
krb5kdc: starting...
W20260924 13:41:57.289986 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:57 dist-test-slave-m5z1 krb5kdc[15488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 13:41:57.406754 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43889
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:57.486491 15501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:57.486688 15501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:57.486711 15501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:57.487877 15501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:57.487921 15501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:57.487936 15501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:57.487996 15501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:57.488014 15501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:57.489497 15501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43889
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:57.489777 15501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:57.490003 15501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:57.495223 15507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:57.495304 15501 server_base.cc:1034] running on GCE node
W20260924 13:41:57.495244 15506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:57.495498 15509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:57.495710 15501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:57.495954 15501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:57.497108 15501 hybrid_clock.cc:648] HybridClock initialized: now 1790257317497083 us; error 42 us; skew 500 ppm
Sep 24 13:41:57 dist-test-slave-m5z1 krb5kdc[15488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:57.499898 15501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 13:41:57.500329 15501 webserver.cc:460] Webserver started at http://127.0.0.1:37973/ using document root <none> and password file <none>
I20260924 13:41:57.500586 15501 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:57.500634 15501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:57.500759 15501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:57.501875 15501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "88d17317f75646c4a4492b530c0234bc"
format_stamp: "Formatted at 2026-09-24 13:41:57 on dist-test-slave-m5z1"
server_key: "bdc3d0145f920c756e3c764eb64a16bf"
server_key_iv: "d24dd407b795f744d4e281f869be6ebe"
server_key_version: "encryptionkey@0"
I20260924 13:41:57.502183 15501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "88d17317f75646c4a4492b530c0234bc"
format_stamp: "Formatted at 2026-09-24 13:41:57 on dist-test-slave-m5z1"
server_key: "bdc3d0145f920c756e3c764eb64a16bf"
server_key_iv: "d24dd407b795f744d4e281f869be6ebe"
server_key_version: "encryptionkey@0"
I20260924 13:41:57.503417 15501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:41:57.504083 15516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:57.504251 15501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:57.504364 15501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "88d17317f75646c4a4492b530c0234bc"
format_stamp: "Formatted at 2026-09-24 13:41:57 on dist-test-slave-m5z1"
server_key: "bdc3d0145f920c756e3c764eb64a16bf"
server_key_iv: "d24dd407b795f744d4e281f869be6ebe"
server_key_version: "encryptionkey@0"
I20260924 13:41:57.504441 15501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:57.507766 15501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:57.510293 15501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:57.510450 15501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:57.514223 15501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43889
I20260924 13:41:57.514251 15568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43889 every 8 connection(s)
I20260924 13:41:57.514573 15501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:57.515529 15569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:57.517833 15569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc: Bootstrap starting.
I20260924 13:41:57.518383 15569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:57.518662 15569 log.cc:824] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:57.519323 15569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc: No bootstrap required, opened a new log
I20260924 13:41:57.520900 15569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d17317f75646c4a4492b530c0234bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } }
I20260924 13:41:57.521031 15569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:57.521083 15569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d17317f75646c4a4492b530c0234bc, State: Initialized, Role: FOLLOWER
I20260924 13:41:57.521261 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15501
I20260924 13:41:57.521243 15569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d17317f75646c4a4492b530c0234bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } }
I20260924 13:41:57.521322 15569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:57.521359 15569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:57.521377 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:57.521411 15569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:57.521603 26561 external_mini_cluster.cc:1428] Setting key 97e9fa3e75b8265f44165c649c603c95
I20260924 13:41:57.522127 15569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d17317f75646c4a4492b530c0234bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } }
I20260924 13:41:57.522279 15569 leader_election.cc:302] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88d17317f75646c4a4492b530c0234bc; no voters:
I20260924 13:41:57.522470 15569 leader_election.cc:288] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:57.522504 15574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:57.522679 15574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [term 1 LEADER]: Becoming Leader. State: Replica: 88d17317f75646c4a4492b530c0234bc, State: Running, Role: LEADER
I20260924 13:41:57.522712 15569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:57.522830 15574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d17317f75646c4a4492b530c0234bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } }
I20260924 13:41:57.524004 15575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88d17317f75646c4a4492b530c0234bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d17317f75646c4a4492b530c0234bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } } }
I20260924 13:41:57.524129 15575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:57.524169 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:57.522058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:34801 (local address 127.0.0.1:43889)
0924 13:41:57.522238 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:57.522247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:57.522280 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:41:57.522395 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:57.522402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:57.522431 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:57.522580 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:57.523490 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:57.523494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:57.523494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:57.523498 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:57.523543 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:57.523557 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:57.523861 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:57.523892 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 13:41:57.523935 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260924 13:41:57.524403 15576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88d17317f75646c4a4492b530c0234bc. Latest consensus state: current_term: 1 leader_uuid: "88d17317f75646c4a4492b530c0234bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d17317f75646c4a4492b530c0234bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43889 } } }
I20260924 13:41:57.524473 15576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:57.525199 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 13:41:57.525286 15590 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:41:57.525341 15590 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:41:57.525422 15587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:57.525580 15587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:57.527057 15587 catalog_manager.cc:1347] Generated new cluster ID: 3077d27cb43441bea8e7425789df6dcc
I20260924 13:41:57.527117 15587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:41:57.527907 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:57.526738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.179:48969 (local address 127.0.0.1:43889)
0924 13:41:57.526767 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:57.526769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:57.526959 (+ 190us) server_negotiation.cc:407] Connection header received
0924 13:41:57.527051 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:57.527052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:57.527056 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:57.527097 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:57.527702 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:57.527703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:57.527703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:41:57.527707 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:57.527718 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:41:57.527722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:57.527749 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:57.527841 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 13:41:57.527853 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 13:41:57.554344 15587 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:57.554535 15587 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:57.558769 15587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88d17317f75646c4a4492b530c0234bc: Generated new TSK 0
I20260924 13:41:57.558972 15587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:57.562481 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15501
2026-09-24T13:41:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:57 dist-test-slave-m5z1 krb5kdc[15600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:57 dist-test-slave-m5z1 krb5kdc[15600](info): set up 2 sockets
Sep 24 13:41:57 dist-test-slave-m5z1 krb5kdc[15600](info): commencing operation
krb5kdc: starting...
W20260924 13:41:58.613869 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:58 dist-test-slave-m5z1 krb5kdc[15600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:41:58.729882 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:45903
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:44847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:45903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:41:58.808481 15613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:41:58.808667 15613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:41:58.808689 15613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:41:58.809758 15613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:41:58.809792 15613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:41:58.809806 15613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:41:58.809819 15613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:41:58.809831 15613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:41:58.811219 15613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:45903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:45903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:41:58.811398 15613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:41:58.811558 15613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:41:58.816238 15619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:58.816246 15621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:41:58.816267 15618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:41:58.816548 15613 server_base.cc:1034] running on GCE node
I20260924 13:41:58.816788 15613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:41:58.817067 15613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:41:58.818229 15613 hybrid_clock.cc:648] HybridClock initialized: now 1790257318818190 us; error 53 us; skew 500 ppm
Sep 24 13:41:58 dist-test-slave-m5z1 krb5kdc[15600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257318, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:41:58.820624 15613 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:41:58.821033 15613 webserver.cc:460] Webserver started at http://127.25.240.126:33785/ using document root <none> and password file <none>
I20260924 13:41:58.821264 15613 fs_manager.cc:359] Metadata directory not provided
I20260924 13:41:58.821332 15613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:41:58.821481 15613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:41:58.822516 15613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "a4ba9c0e5a23497a8df052a8a5475039"
format_stamp: "Formatted at 2026-09-24 13:41:58 on dist-test-slave-m5z1"
server_key: "376ab8a76fe583f12fa18680b026f969"
server_key_iv: "63ed47b4b3278a9075f1d94476596782"
server_key_version: "encryptionkey@0"
I20260924 13:41:58.822867 15613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "a4ba9c0e5a23497a8df052a8a5475039"
format_stamp: "Formatted at 2026-09-24 13:41:58 on dist-test-slave-m5z1"
server_key: "376ab8a76fe583f12fa18680b026f969"
server_key_iv: "63ed47b4b3278a9075f1d94476596782"
server_key_version: "encryptionkey@0"
I20260924 13:41:58.824119 15613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:41:58.824787 15628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:41:58.824949 15613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:41:58.825052 15613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "a4ba9c0e5a23497a8df052a8a5475039"
format_stamp: "Formatted at 2026-09-24 13:41:58 on dist-test-slave-m5z1"
server_key: "376ab8a76fe583f12fa18680b026f969"
server_key_iv: "63ed47b4b3278a9075f1d94476596782"
server_key_version: "encryptionkey@0"
I20260924 13:41:58.825129 15613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:41:58.834800 15613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:41:58.836441 15613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:41:58.836596 15613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:41:58.840207 15613 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:45903
I20260924 13:41:58.840247 15680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:45903 every 8 connection(s)
I20260924 13:41:58.840557 15613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:41:58.841339 15681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:41:58.843457 15681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039: Bootstrap starting.
I20260924 13:41:58.843915 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15613
I20260924 13:41:58.844026 15681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039: Neither blocks nor log segments found. Creating new log.
I20260924 13:41:58.844048 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:41:58.844241 26561 external_mini_cluster.cc:1428] Setting key 1d40928d45cfa9db058bacaa9a0cd343
I20260924 13:41:58.844327 15681 log.cc:824] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039: Log is configured to *not* fsync() on all Append() calls
I20260924 13:41:58.845078 15681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039: No bootstrap required, opened a new log
I20260924 13:41:58.847026 15681 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ba9c0e5a23497a8df052a8a5475039" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45903 } }
I20260924 13:41:58.847187 15681 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:41:58.847209 15681 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ba9c0e5a23497a8df052a8a5475039, State: Initialized, Role: FOLLOWER
I20260924 13:41:58.847381 15681 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ba9c0e5a23497a8df052a8a5475039" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45903 } }
I20260924 13:41:58.847453 15681 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:41:58.847473 15681 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:41:58.847532 15681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:41:58.848035 15681 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ba9c0e5a23497a8df052a8a5475039" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45903 } }
I20260924 13:41:58.848148 15681 leader_election.cc:302] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4ba9c0e5a23497a8df052a8a5475039; no voters:
I20260924 13:41:58.848315 15681 leader_election.cc:288] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:41:58.848335 15686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:41:58.848448 15686 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [term 1 LEADER]: Becoming Leader. State: Replica: a4ba9c0e5a23497a8df052a8a5475039, State: Running, Role: LEADER
I20260924 13:41:58.848572 15681 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:41:58.848593 15686 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ba9c0e5a23497a8df052a8a5475039" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45903 } }
I20260924 13:41:58.849012 15688 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4ba9c0e5a23497a8df052a8a5475039. Latest consensus state: current_term: 1 leader_uuid: "a4ba9c0e5a23497a8df052a8a5475039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ba9c0e5a23497a8df052a8a5475039" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45903 } } }
I20260924 13:41:58.849098 15688 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:58.849303 15693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:41:58.849336 15687 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4ba9c0e5a23497a8df052a8a5475039" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4ba9c0e5a23497a8df052a8a5475039" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 45903 } } }
I20260924 13:41:58.849611 15687 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039 [sys.catalog]: This master's current role is: LEADER
I20260924 13:41:58.849849 15693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:41:58.851389 15693 catalog_manager.cc:1347] Generated new cluster ID: 70be25c120c04e2399d76a8598428b37
I20260924 13:41:58.851440 15693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 13:41:58 dist-test-slave-m5z1 krb5kdc[15600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:41:58.855712 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:58.844742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53973 (local address 127.25.240.126:45903)
0924 13:41:58.844878 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:58.844888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:58.844921 (+ 33us) server_negotiation.cc:407] Connection header received
0924 13:41:58.845394 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:58.845401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:58.845443 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:58.845604 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:58.846235 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:58.847118 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:58.848035 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:58.848652 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:58.849064 (+ 412us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:58.852347 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:58.852351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:58.852351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:58.852362 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:58.854079 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:58.854513 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:58.854516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:58.854516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:58.854609 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:58.854853 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:58.854854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:58.854854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:58.855137 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:58.855267 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:58.855434 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 13:41:58.855520 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260924 13:41:58.856561 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:41:58.862284 15693 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:41:58.862933 15693 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:41:58.866122 15693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4ba9c0e5a23497a8df052a8a5475039: Generated new TSK 0
I20260924 13:41:58.866285 15693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:41:58.869140 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:41:58.861892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53241 (local address 127.25.240.126:45903)
0924 13:41:58.861930 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:41:58.861932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:41:58.861988 (+ 56us) server_negotiation.cc:407] Connection header received
0924 13:41:58.862068 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:41:58.862069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:41:58.862079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:41:58.862129 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:41:58.862740 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:58.863530 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:58.864507 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:41:58.864839 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:41:58.864910 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:41:58.866315 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:41:58.866317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:41:58.866317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:41:58.866323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:41:58.867789 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:58.868144 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:58.868145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:58.868146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:58.868205 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:41:58.868407 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:41:58.868407 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:41:58.868408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:41:58.868530 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:41:58.868729 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 13:41:58.868973 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 13:41:58.869040 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":20}
I20260924 13:41:58.871680 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15613
W20260924 13:41:58.875531 15598 connection.cc:535] client connection to 127.25.240.126:45903 recv error: Network error: failed to read from TLS socket (remote: 127.25.240.126:45903): Connection reset by peer (error 104)
2026-09-24T13:41:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:41:58 dist-test-slave-m5z1 krb5kdc[15712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:41:58 dist-test-slave-m5z1 krb5kdc[15712](info): set up 2 sockets
Sep 24 13:41:58 dist-test-slave-m5z1 krb5kdc[15712](info): commencing operation
krb5kdc: starting...
W20260924 13:41:59.890004 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:41:59 dist-test-slave-m5z1 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:41:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:41:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:42:00.003793 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37095
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:37095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:00.081736 15725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:00.081960 15725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:00.081996 15725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:00.083110 15725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:00.083180 15725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:00.083205 15725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:00.083232 15725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:00.083264 15725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:00.084703 15725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:37095
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:00.084970 15725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:00.085193 15725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:00.090055 15730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:00.090135 15731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:00.090170 15725 server_base.cc:1034] running on GCE node
W20260924 13:42:00.090059 15733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:00.090452 15725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:00.090698 15725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:00.091835 15725 hybrid_clock.cc:648] HybridClock initialized: now 1790257320091828 us; error 31 us; skew 500 ppm
Sep 24 13:42:00 dist-test-slave-m5z1 krb5kdc[15712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257320, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:00.116823 15725 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:42:00.117377 15725 webserver.cc:460] Webserver started at http://127.25.240.126:39499/ using document root <none> and password file <none>
I20260924 13:42:00.117695 15725 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:00.117748 15725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:00.117873 15725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:00.119004 15725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "4d01ee11182641839db40d677da7b893"
format_stamp: "Formatted at 2026-09-24 13:42:00 on dist-test-slave-m5z1"
server_key: "afacb393e45e6a3801e2bffe4963e662"
server_key_iv: "4e7778f2a1e5f914f5c761ff6e3c079b"
server_key_version: "encryptionkey@0"
I20260924 13:42:00.119376 15725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "4d01ee11182641839db40d677da7b893"
format_stamp: "Formatted at 2026-09-24 13:42:00 on dist-test-slave-m5z1"
server_key: "afacb393e45e6a3801e2bffe4963e662"
server_key_iv: "4e7778f2a1e5f914f5c761ff6e3c079b"
server_key_version: "encryptionkey@0"
I20260924 13:42:00.120743 15725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 13:42:00.121618 15740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:00.121804 15725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 13:42:00.121893 15725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "4d01ee11182641839db40d677da7b893"
format_stamp: "Formatted at 2026-09-24 13:42:00 on dist-test-slave-m5z1"
server_key: "afacb393e45e6a3801e2bffe4963e662"
server_key_iv: "4e7778f2a1e5f914f5c761ff6e3c079b"
server_key_version: "encryptionkey@0"
I20260924 13:42:00.121948 15725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:00.171715 15725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:00.173729 15725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:00.173878 15725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:00.177603 15725 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:37095
I20260924 13:42:00.177634 15792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:37095 every 8 connection(s)
I20260924 13:42:00.177999 15725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:00.178774 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15725
I20260924 13:42:00.178867 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:42:00.178963 15793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:00.179103 26561 external_mini_cluster.cc:1428] Setting key 858699b9ce7440122bc895d46349cc48
I20260924 13:42:00.182497 15793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893: Bootstrap starting.
I20260924 13:42:00.183187 15793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:00.183490 15793 log.cc:824] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:00.184334 15793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893: No bootstrap required, opened a new log
I20260924 13:42:00.185072 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:00.179649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46385 (local address 127.25.240.126:37095)
0924 13:42:00.179847 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:00.179857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:00.179893 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:42:00.180301 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:00.180309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:00.180382 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:00.180534 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:00.180690 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:00.182075 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:00.182854 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:00.183492 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:00.183602 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:00.184362 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:00.184367 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:00.184367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:00.184574 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:00.184611 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:00.184625 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:00.184714 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:00.184791 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 13:42:00.184819 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
I20260924 13:42:00.186132 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:00.186906 15793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d01ee11182641839db40d677da7b893" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37095 } }
I20260924 13:42:00.187083 15793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:00.187120 15793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d01ee11182641839db40d677da7b893, State: Initialized, Role: FOLLOWER
I20260924 13:42:00.187271 15793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d01ee11182641839db40d677da7b893" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37095 } }
I20260924 13:42:00.187368 15793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:00.187427 15793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:00.187480 15793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:00.188277 15793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d01ee11182641839db40d677da7b893" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37095 } }
I20260924 13:42:00.188367 15793 leader_election.cc:302] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d01ee11182641839db40d677da7b893; no voters:
I20260924 13:42:00.188563 15793 leader_election.cc:288] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:00.188593 15803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:00.188710 15803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [term 1 LEADER]: Becoming Leader. State: Replica: 4d01ee11182641839db40d677da7b893, State: Running, Role: LEADER
I20260924 13:42:00.188817 15793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:00.188797 15803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d01ee11182641839db40d677da7b893" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37095 } }
I20260924 13:42:00.189150 15803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d01ee11182641839db40d677da7b893. Latest consensus state: current_term: 1 leader_uuid: "4d01ee11182641839db40d677da7b893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d01ee11182641839db40d677da7b893" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37095 } } }
I20260924 13:42:00.189224 15803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:00.189605 15804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d01ee11182641839db40d677da7b893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d01ee11182641839db40d677da7b893" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37095 } } }
I20260924 13:42:00.189677 15804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:00.190059 15809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:00.190889 15809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:00.190981 15796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:00.187677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55835 (local address 127.25.240.126:37095)
0924 13:42:00.187739 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:00.187741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:00.187787 (+ 46us) server_negotiation.cc:407] Connection header received
0924 13:42:00.187862 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:00.187863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:00.187870 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:00.187913 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:00.188036 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:00.188702 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:00.189716 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:00.190025 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:00.190101 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:00.190700 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:00.190701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:00.190702 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:00.190706 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:00.190719 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:00.190723 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:00.190774 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:00.190831 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 13:42:00.190840 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 13:42:00.192292 15809 catalog_manager.cc:1347] Generated new cluster ID: 4c8c4041ecb249bb9c573a12691e3d8e
I20260924 13:42:00.192344 15809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:00.207664 15809 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:00.208209 15809 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:00.215788 15809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d01ee11182641839db40d677da7b893: Generated new TSK 0
I20260924 13:42:00.215967 15809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:00.221661 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15725
W20260924 13:42:00.226398 15710 connection.cc:535] client connection to 127.25.240.126:37095 recv error: Network error: failed to read from TLS socket (remote: 127.25.240.126:37095): Connection reset by peer (error 104)
2026-09-24T13:42:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:00 dist-test-slave-m5z1 krb5kdc[15824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:00 dist-test-slave-m5z1 krb5kdc[15824](info): set up 2 sockets
Sep 24 13:42:00 dist-test-slave-m5z1 krb5kdc[15824](info): commencing operation
krb5kdc: starting...
W20260924 13:42:01.253934 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:42:01 dist-test-slave-m5z1 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:42:01.372493 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:46677
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:34267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:46677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:01.453178 15837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:01.453409 15837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:01.453445 15837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:01.454584 15837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:01.454650 15837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:01.454674 15837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:01.454694 15837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:01.454725 15837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:01.456175 15837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:34267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:46677
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:46677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:01.456452 15837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:01.456677 15837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:01.461603 15837 server_base.cc:1034] running on GCE node
W20260924 13:42:01.461527 15842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:01.461548 15843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:01.461527 15845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:01.462054 15837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:01.462360 15837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:01.463522 15837 hybrid_clock.cc:648] HybridClock initialized: now 1790257321463513 us; error 44 us; skew 500 ppm
Sep 24 13:42:01 dist-test-slave-m5z1 krb5kdc[15824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257321, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:01.466007 15837 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:42:01.466372 15837 webserver.cc:460] Webserver started at http://127.25.240.126:39267/ using document root <none> and password file <none>
I20260924 13:42:01.466611 15837 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:01.466681 15837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:01.466832 15837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:01.467875 15837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "cdc4cb5504274d15bdba9de7302a49b6"
format_stamp: "Formatted at 2026-09-24 13:42:01 on dist-test-slave-m5z1"
server_key: "6e34d017d85b41cbbfe1254a016b9185"
server_key_iv: "f46191f56a916683a678662c021393c5"
server_key_version: "encryptionkey@0"
I20260924 13:42:01.468240 15837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "cdc4cb5504274d15bdba9de7302a49b6"
format_stamp: "Formatted at 2026-09-24 13:42:01 on dist-test-slave-m5z1"
server_key: "6e34d017d85b41cbbfe1254a016b9185"
server_key_iv: "f46191f56a916683a678662c021393c5"
server_key_version: "encryptionkey@0"
I20260924 13:42:01.469528 15837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:01.470295 15852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:01.470476 15837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:01.470562 15837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "cdc4cb5504274d15bdba9de7302a49b6"
format_stamp: "Formatted at 2026-09-24 13:42:01 on dist-test-slave-m5z1"
server_key: "6e34d017d85b41cbbfe1254a016b9185"
server_key_iv: "f46191f56a916683a678662c021393c5"
server_key_version: "encryptionkey@0"
I20260924 13:42:01.470628 15837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:01.473790 15837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:01.476001 15837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:01.476150 15837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:01.480031 15837 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:46677
I20260924 13:42:01.480058 15904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:46677 every 8 connection(s)
I20260924 13:42:01.480381 15837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:01.481331 15905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:01.483523 15905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6: Bootstrap starting.
I20260924 13:42:01.484082 15905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:01.484357 15905 log.cc:824] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:01.485219 15905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6: No bootstrap required, opened a new log
I20260924 13:42:01.486865 15905 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc4cb5504274d15bdba9de7302a49b6" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46677 } }
I20260924 13:42:01.487023 15905 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:01.487062 15905 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc4cb5504274d15bdba9de7302a49b6, State: Initialized, Role: FOLLOWER
I20260924 13:42:01.487085 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15837
I20260924 13:42:01.487192 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:42:01.487259 15905 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc4cb5504274d15bdba9de7302a49b6" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46677 } }
I20260924 13:42:01.487349 15905 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:01.487390 15905 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:01.487416 26561 external_mini_cluster.cc:1428] Setting key 441efa3df2716be195cb0f602b41bbaf
I20260924 13:42:01.487440 15905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:01.488519 15905 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc4cb5504274d15bdba9de7302a49b6" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46677 } }
I20260924 13:42:01.488631 15905 leader_election.cc:302] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc4cb5504274d15bdba9de7302a49b6; no voters:
I20260924 13:42:01.488804 15905 leader_election.cc:288] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:01.488827 15910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:01.488957 15910 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [term 1 LEADER]: Becoming Leader. State: Replica: cdc4cb5504274d15bdba9de7302a49b6, State: Running, Role: LEADER
I20260924 13:42:01.489094 15905 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:01.489063 15910 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc4cb5504274d15bdba9de7302a49b6" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46677 } }
I20260924 13:42:01.489866 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:01.487937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50461 (local address 127.25.240.126:46677)
0924 13:42:01.488121 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:01.488128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:01.488153 (+ 25us) server_negotiation.cc:407] Connection header received
0924 13:42:01.488242 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:01.488247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:01.488268 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:01.488382 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:01.489159 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:01.489163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:01.489163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:01.489168 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:01.489238 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:01.489253 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:01.489312 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:01.489587 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 13:42:01.489632 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260924 13:42:01.489892 15911 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc4cb5504274d15bdba9de7302a49b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc4cb5504274d15bdba9de7302a49b6" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46677 } } }
I20260924 13:42:01.489969 15911 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:01.490121 15912 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc4cb5504274d15bdba9de7302a49b6. Latest consensus state: current_term: 1 leader_uuid: "cdc4cb5504274d15bdba9de7302a49b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc4cb5504274d15bdba9de7302a49b6" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 46677 } } }
I20260924 13:42:01.490473 15912 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:01.490983 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 13:42:01.491057 15926 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:01.491113 15926 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:01.491204 15919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:01.491380 15919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:01.493103 15919 catalog_manager.cc:1347] Generated new cluster ID: c87501295c9b44caad7aa37570540724
I20260924 13:42:01.493161 15919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:01.493741 15909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:01.492551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39279 (local address 127.25.240.126:46677)
0924 13:42:01.492587 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:01.492588 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:01.492724 (+ 136us) server_negotiation.cc:407] Connection header received
0924 13:42:01.492778 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:01.492779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:01.492784 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:01.492841 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:01.493480 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:01.493481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:01.493482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:01.493485 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:01.493498 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:01.493502 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:01.493543 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:01.493667 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 13:42:01.493680 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:01.518070 15919 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:01.518271 15919 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:01.533735 15919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc4cb5504274d15bdba9de7302a49b6: Generated new TSK 0
I20260924 13:42:01.533959 15919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:01.539245 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15837
2026-09-24T13:42:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:01 dist-test-slave-m5z1 krb5kdc[15936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:01 dist-test-slave-m5z1 krb5kdc[15936](info): set up 2 sockets
Sep 24 13:42:01 dist-test-slave-m5z1 krb5kdc[15936](info): commencing operation
krb5kdc: starting...
W20260924 13:42:02.577977 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:42:02 dist-test-slave-m5z1 krb5kdc[15936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:42:02.696595 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:44811
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:44811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:02.778045 15949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:02.778298 15949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:02.778321 15949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:02.779424 15949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:02.779458 15949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:02.779472 15949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:02.779485 15949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:02.779497 15949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:02.780850 15949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:44811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:44811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:02.781044 15949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:02.781200 15949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:02.785945 15955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:02.785955 15954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:02.786012 15957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:02.786123 15949 server_base.cc:1034] running on GCE node
I20260924 13:42:02.786340 15949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:02.786564 15949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:02.787736 15949 hybrid_clock.cc:648] HybridClock initialized: now 1790257322787710 us; error 38 us; skew 500 ppm
Sep 24 13:42:02 dist-test-slave-m5z1 krb5kdc[15936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257322, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:02.790120 15949 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:42:02.790530 15949 webserver.cc:460] Webserver started at http://127.25.240.126:42015/ using document root <none> and password file <none>
I20260924 13:42:02.790791 15949 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:02.790860 15949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:02.791002 15949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:02.792029 15949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "b3106e4ab2164c798cb2ae1835748238"
format_stamp: "Formatted at 2026-09-24 13:42:02 on dist-test-slave-m5z1"
server_key: "9f763dd0360a030e480b646cd75f52a0"
server_key_iv: "14b053eaa5b7765d6869760dc88df8b3"
server_key_version: "encryptionkey@0"
I20260924 13:42:02.792375 15949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "b3106e4ab2164c798cb2ae1835748238"
format_stamp: "Formatted at 2026-09-24 13:42:02 on dist-test-slave-m5z1"
server_key: "9f763dd0360a030e480b646cd75f52a0"
server_key_iv: "14b053eaa5b7765d6869760dc88df8b3"
server_key_version: "encryptionkey@0"
I20260924 13:42:02.793717 15949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 13:42:02.794418 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:02.794601 15949 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:02.794690 15949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "b3106e4ab2164c798cb2ae1835748238"
format_stamp: "Formatted at 2026-09-24 13:42:02 on dist-test-slave-m5z1"
server_key: "9f763dd0360a030e480b646cd75f52a0"
server_key_iv: "14b053eaa5b7765d6869760dc88df8b3"
server_key_version: "encryptionkey@0"
I20260924 13:42:02.794770 15949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:02.819797 15949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:02.821703 15949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:02.821859 15949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:02.825534 15949 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:44811
I20260924 13:42:02.825613 16016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:44811 every 8 connection(s)
I20260924 13:42:02.825996 15949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:02.826840 16017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:02.829173 16017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238: Bootstrap starting.
I20260924 13:42:02.829767 16017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:02.830049 16017 log.cc:824] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:02.830705 16017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238: No bootstrap required, opened a new log
I20260924 13:42:02.831352 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 15949
I20260924 13:42:02.831477 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:42:02.831666 26561 external_mini_cluster.cc:1428] Setting key b55c17fa1c20292462214e46fd75788a
I20260924 13:42:02.833325 16017 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3106e4ab2164c798cb2ae1835748238" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44811 } }
I20260924 13:42:02.833506 16017 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:02.833550 16017 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3106e4ab2164c798cb2ae1835748238, State: Initialized, Role: FOLLOWER
I20260924 13:42:02.833719 16017 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3106e4ab2164c798cb2ae1835748238" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44811 } }
I20260924 13:42:02.833806 16017 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:02.833839 16017 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:02.833889 16017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:02.834537 16017 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3106e4ab2164c798cb2ae1835748238" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44811 } }
I20260924 13:42:02.834664 16017 leader_election.cc:302] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3106e4ab2164c798cb2ae1835748238; no voters:
I20260924 13:42:02.834827 16017 leader_election.cc:288] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:02.834874 16022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:02.835088 16022 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [term 1 LEADER]: Becoming Leader. State: Replica: b3106e4ab2164c798cb2ae1835748238, State: Running, Role: LEADER
I20260924 13:42:02.835132 16017 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:02.836488 16022 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3106e4ab2164c798cb2ae1835748238" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44811 } }
I20260924 13:42:02.837054 16023 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3106e4ab2164c798cb2ae1835748238. Latest consensus state: current_term: 1 leader_uuid: "b3106e4ab2164c798cb2ae1835748238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3106e4ab2164c798cb2ae1835748238" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44811 } } }
I20260924 13:42:02.837491 16023 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:02.837186 16024 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3106e4ab2164c798cb2ae1835748238" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3106e4ab2164c798cb2ae1835748238" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 44811 } } }
I20260924 13:42:02.837664 16024 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238 [sys.catalog]: This master's current role is: LEADER
W20260924 13:42:02.837877 16036 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:02.837934 16036 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:02.837984 16038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:02.838179 16038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 13:42:02 dist-test-slave-m5z1 krb5kdc[15936](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:42:02.839583 16038 catalog_manager.cc:1347] Generated new cluster ID: b9d4001d492c45a0a11df806e8211a0d
I20260924 13:42:02.839632 16038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:02.843211 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:02.832177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44857 (local address 127.25.240.126:44811)
0924 13:42:02.832334 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:02.832345 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:02.832379 (+ 34us) server_negotiation.cc:407] Connection header received
0924 13:42:02.833260 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:02.833266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:02.833314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:02.833425 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:02.834288 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:02.835615 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:02.836523 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:02.837136 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:02.837258 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:02.839956 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:02.839960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:02.839960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:02.839972 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:02.841662 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:02.842031 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:02.842034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:02.842035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:02.842134 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:02.842340 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:02.842341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:02.842341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:02.842621 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:02.842761 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:02.842880 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 13:42:02.843009 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260924 13:42:02.844045 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:02.852478 16021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:02.845515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33675 (local address 127.25.240.126:44811)
0924 13:42:02.845553 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:02.845575 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:02.845745 (+ 170us) server_negotiation.cc:407] Connection header received
0924 13:42:02.845770 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:02.845771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:02.845781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:02.845848 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:02.846392 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:02.847153 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:02.848112 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:02.848430 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:02.848514 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:02.849811 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:02.849812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:02.849813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:02.849819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:02.851350 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:02.851723 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:02.851724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:02.851724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:02.851784 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:02.851973 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:02.851974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:02.851974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:02.852102 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:02.852196 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:02.852298 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 13:42:02.852371 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:02.865022 16038 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:02.865787 16038 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:02.878789 16038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3106e4ab2164c798cb2ae1835748238: Generated new TSK 0
I20260924 13:42:02.878984 16038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:02.884411 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 15949
2026-09-24T13:42:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:02 dist-test-slave-m5z1 krb5kdc[16048](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:02 dist-test-slave-m5z1 krb5kdc[16048](info): set up 2 sockets
Sep 24 13:42:02 dist-test-slave-m5z1 krb5kdc[16048](info): commencing operation
krb5kdc: starting...
W20260924 13:42:03.918005 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:42:04 dist-test-slave-m5z1 krb5kdc[16048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:42:04.035782 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38545
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:38545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:04.115444 16061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:04.115639 16061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:04.115662 16061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:04.116710 16061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:04.116745 16061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:04.116760 16061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:04.116773 16061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:04.116786 16061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:04.118146 16061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:38545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:38545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:04.118321 16061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:04.118477 16061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:04.123399 16067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:04.123399 16066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:04.123644 16069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:04.123617 16061 server_base.cc:1034] running on GCE node
I20260924 13:42:04.123869 16061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:04.124114 16061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:04.125252 16061 hybrid_clock.cc:648] HybridClock initialized: now 1790257324125234 us; error 34 us; skew 500 ppm
Sep 24 13:42:04 dist-test-slave-m5z1 krb5kdc[16048](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257324, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:04.127771 16061 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:42:04.128230 16061 webserver.cc:460] Webserver started at http://127.25.240.126:41267/ using document root <none> and password file <none>
I20260924 13:42:04.128456 16061 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:04.128527 16061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:04.128676 16061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:04.129859 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "ce20c812f8d241cda395c11b0f5a9be3"
format_stamp: "Formatted at 2026-09-24 13:42:04 on dist-test-slave-m5z1"
server_key: "0b51eb544d3e50ba952abec09c483ff7"
server_key_iv: "4b10eb153f90b69c5ed1927ad2df9297"
server_key_version: "encryptionkey@0"
I20260924 13:42:04.130229 16061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "ce20c812f8d241cda395c11b0f5a9be3"
format_stamp: "Formatted at 2026-09-24 13:42:04 on dist-test-slave-m5z1"
server_key: "0b51eb544d3e50ba952abec09c483ff7"
server_key_iv: "4b10eb153f90b69c5ed1927ad2df9297"
server_key_version: "encryptionkey@0"
I20260924 13:42:04.131559 16061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:04.132351 16076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:04.132503 16061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:04.132591 16061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "ce20c812f8d241cda395c11b0f5a9be3"
format_stamp: "Formatted at 2026-09-24 13:42:04 on dist-test-slave-m5z1"
server_key: "0b51eb544d3e50ba952abec09c483ff7"
server_key_iv: "4b10eb153f90b69c5ed1927ad2df9297"
server_key_version: "encryptionkey@0"
I20260924 13:42:04.132645 16061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:04.151815 16061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:04.153693 16061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:04.153802 16061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:04.157464 16061 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:38545
I20260924 13:42:04.157522 16128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:38545 every 8 connection(s)
I20260924 13:42:04.157858 16061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:04.158725 16129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:04.160255 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 16061
I20260924 13:42:04.160346 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:42:04.160531 26561 external_mini_cluster.cc:1428] Setting key 217bc17e67147a90bf0094eab66215dd
I20260924 13:42:04.161130 16129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3: Bootstrap starting.
I20260924 13:42:04.161749 16129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:04.162103 16129 log.cc:824] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:04.162807 16129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3: No bootstrap required, opened a new log
I20260924 13:42:04.165108 16129 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce20c812f8d241cda395c11b0f5a9be3" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } }
I20260924 13:42:04.165282 16129 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:04.165318 16129 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce20c812f8d241cda395c11b0f5a9be3, State: Initialized, Role: FOLLOWER
I20260924 13:42:04.165442 16129 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce20c812f8d241cda395c11b0f5a9be3" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } }
I20260924 13:42:04.165544 16129 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:04.165616 16129 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:04.165669 16129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:04.166155 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:04.161044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52519 (local address 127.25.240.126:38545)
0924 13:42:04.161234 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:04.161244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:04.161280 (+ 36us) server_negotiation.cc:407] Connection header received
0924 13:42:04.161773 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:04.161779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:04.161826 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:04.161976 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:04.162243 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:04.163186 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:04.164296 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:04.164683 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:04.164760 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:04.165425 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:04.165429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:04.165429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:04.165660 (+ 231us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:04.165701 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:04.165715 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:04.165780 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:04.165864 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 13:42:04.165933 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260924 13:42:04.166301 16129 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce20c812f8d241cda395c11b0f5a9be3" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } }
I20260924 13:42:04.166435 16129 leader_election.cc:302] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce20c812f8d241cda395c11b0f5a9be3; no voters:
I20260924 13:42:04.166594 16129 leader_election.cc:288] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:04.166647 16134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:04.166790 16134 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [term 1 LEADER]: Becoming Leader. State: Replica: ce20c812f8d241cda395c11b0f5a9be3, State: Running, Role: LEADER
I20260924 13:42:04.166843 16129 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:04.166947 16134 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce20c812f8d241cda395c11b0f5a9be3" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } }
I20260924 13:42:04.167133 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:04.167266 16134 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce20c812f8d241cda395c11b0f5a9be3. Latest consensus state: current_term: 1 leader_uuid: "ce20c812f8d241cda395c11b0f5a9be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce20c812f8d241cda395c11b0f5a9be3" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } } }
I20260924 13:42:04.167321 16134 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:04.167447 16136 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce20c812f8d241cda395c11b0f5a9be3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce20c812f8d241cda395c11b0f5a9be3" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 38545 } } }
I20260924 13:42:04.167485 16136 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3 [sys.catalog]: This master's current role is: LEADER
W20260924 13:42:04.168640 16151 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:04.168699 16151 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:04.168804 16141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:04.169030 16141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:04.170432 16141 catalog_manager.cc:1347] Generated new cluster ID: 2067c23ba6464ee7a1f2273a7137b209
I20260924 13:42:04.170485 16141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:04.172410 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:04.169005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33935 (local address 127.25.240.126:38545)
0924 13:42:04.169042 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:04.169043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:04.169183 (+ 140us) server_negotiation.cc:407] Connection header received
0924 13:42:04.169203 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:04.169204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:04.169214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:04.169256 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:04.169407 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:04.170138 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:04.171148 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:04.171453 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:04.171525 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:04.172182 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:04.172183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:04.172183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:04.172187 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:04.172202 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:04.172206 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:04.172255 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:04.172320 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 13:42:04.172340 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:04.200230 16141 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:04.200979 16141 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:04.216534 16141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce20c812f8d241cda395c11b0f5a9be3: Generated new TSK 0
I20260924 13:42:04.216751 16141 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:04.230484 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 16061
2026-09-24T13:42:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:04 dist-test-slave-m5z1 krb5kdc[16160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:04 dist-test-slave-m5z1 krb5kdc[16160](info): set up 2 sockets
Sep 24 13:42:04 dist-test-slave-m5z1 krb5kdc[16160](info): commencing operation
krb5kdc: starting...
W20260924 13:42:05.257931 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:42:05 dist-test-slave-m5z1 krb5kdc[16160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:42:05.372778 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:41139
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:38367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:41139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:05.452253 16173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:05.452495 16173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:05.452539 16173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:05.453692 16173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:05.453756 16173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:05.453778 16173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:05.453800 16173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:05.453826 16173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:05.455286 16173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:38367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:41139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:41139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:05.455554 16173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:05.455775 16173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:05.460614 16178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:05.460630 16173 server_base.cc:1034] running on GCE node
W20260924 13:42:05.460602 16181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:05.460602 16179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:05.461099 16173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:05.461385 16173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:05.462548 16173 hybrid_clock.cc:648] HybridClock initialized: now 1790257325462506 us; error 51 us; skew 500 ppm
Sep 24 13:42:05 dist-test-slave-m5z1 krb5kdc[16160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257325, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:05.470876 16173 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:42:05.471315 16173 webserver.cc:460] Webserver started at http://127.25.240.126:40235/ using document root <none> and password file <none>
I20260924 13:42:05.471566 16173 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:05.471642 16173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:05.471779 16173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:05.472826 16173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "dc7cc6a925c145189fa786aabef438ff"
format_stamp: "Formatted at 2026-09-24 13:42:05 on dist-test-slave-m5z1"
server_key: "7e9de27ecf77c95aeed459d2f1174633"
server_key_iv: "4df52e8f846d3aba23242666290bbf4e"
server_key_version: "encryptionkey@0"
I20260924 13:42:05.473167 16173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "dc7cc6a925c145189fa786aabef438ff"
format_stamp: "Formatted at 2026-09-24 13:42:05 on dist-test-slave-m5z1"
server_key: "7e9de27ecf77c95aeed459d2f1174633"
server_key_iv: "4df52e8f846d3aba23242666290bbf4e"
server_key_version: "encryptionkey@0"
I20260924 13:42:05.474531 16173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 13:42:05.475257 16188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:05.475419 16173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:05.475518 16173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "dc7cc6a925c145189fa786aabef438ff"
format_stamp: "Formatted at 2026-09-24 13:42:05 on dist-test-slave-m5z1"
server_key: "7e9de27ecf77c95aeed459d2f1174633"
server_key_iv: "4df52e8f846d3aba23242666290bbf4e"
server_key_version: "encryptionkey@0"
I20260924 13:42:05.475600 16173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:05.478884 16173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:05.480636 16173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:05.480782 16173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:05.484613 16173 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:41139
I20260924 13:42:05.484642 16240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:41139 every 8 connection(s)
I20260924 13:42:05.484952 16173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:05.485930 16241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:05.487074 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 16173
I20260924 13:42:05.487206 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:42:05.487380 26561 external_mini_cluster.cc:1428] Setting key 54b7c854e55de370c4fe73f8db3d6c19
I20260924 13:42:05.488605 16241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff: Bootstrap starting.
I20260924 13:42:05.489269 16241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:05.489594 16241 log.cc:824] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:05.489880 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:05.487860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36197 (local address 127.25.240.126:41139)
0924 13:42:05.488041 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:05.488050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:05.488088 (+ 38us) server_negotiation.cc:407] Connection header received
0924 13:42:05.488614 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:05.488623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:05.488656 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:05.488807 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:05.489423 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:05.489427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:05.489427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:05.489431 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:05.489465 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:05.489475 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:05.489559 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:05.489646 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 13:42:05.489687 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260924 13:42:05.490556 16241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff: No bootstrap required, opened a new log
I20260924 13:42:05.490823 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:05.492503 16241 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cc6a925c145189fa786aabef438ff" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41139 } }
I20260924 13:42:05.492651 16241 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 13:42:05.492697 16241 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7cc6a925c145189fa786aabef438ff, State: Initialized, Role: FOLLOWER
I20260924 13:42:05.492813 16241 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cc6a925c145189fa786aabef438ff" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41139 } }
I20260924 13:42:05.492913 16241 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 13:42:05.492959 16241 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 13:42:05.493006 16241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:05.493207 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:05.492161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46003 (local address 127.25.240.126:41139)
0924 13:42:05.492196 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:05.492198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:05.492312 (+ 114us) server_negotiation.cc:407] Connection header received
0924 13:42:05.492381 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:05.492382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:05.492387 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:05.492440 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:05.493003 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:05.493004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:05.493005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 13:42:05.493009 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:05.493022 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 13:42:05.493026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:05.493075 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:05.493131 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 13:42:05.493145 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 13:42:05.493755 16241 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cc6a925c145189fa786aabef438ff" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41139 } }
I20260924 13:42:05.493856 16241 leader_election.cc:302] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc7cc6a925c145189fa786aabef438ff; no voters:
I20260924 13:42:05.494022 16241 leader_election.cc:288] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 13:42:05.494041 16251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:05.494221 16251 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [term 1 LEADER]: Becoming Leader. State: Replica: dc7cc6a925c145189fa786aabef438ff, State: Running, Role: LEADER
I20260924 13:42:05.494253 16241 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:05.494365 16251 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cc6a925c145189fa786aabef438ff" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41139 } }
I20260924 13:42:05.494794 16253 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc7cc6a925c145189fa786aabef438ff. Latest consensus state: current_term: 1 leader_uuid: "dc7cc6a925c145189fa786aabef438ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cc6a925c145189fa786aabef438ff" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41139 } } }
I20260924 13:42:05.494781 16252 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc7cc6a925c145189fa786aabef438ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc7cc6a925c145189fa786aabef438ff" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41139 } } }
I20260924 13:42:05.494956 16253 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:05.494979 16252 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:05.495417 16260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:05.495878 16260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:05.497372 16260 catalog_manager.cc:1347] Generated new cluster ID: 382a85016b6c4ef994bc581dc91636c6
I20260924 13:42:05.497434 16260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:05.516852 16260 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:05.517096 16260 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:05.530099 16260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc7cc6a925c145189fa786aabef438ff: Generated new TSK 0
I20260924 13:42:05.530295 16260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 13:42:05.534973 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 16173
2026-09-24T13:42:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1304 ms)
[----------] 18 tests from AuthTokenIssuingTest (23778 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:05 dist-test-slave-m5z1 krb5kdc[16272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:05 dist-test-slave-m5z1 krb5kdc[16272](info): set up 2 sockets
Sep 24 13:42:05 dist-test-slave-m5z1 krb5kdc[16272](info): commencing operation
krb5kdc: starting...
W20260924 13:42:06.573936 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:42:06 dist-test-slave-m5z1 krb5kdc[16272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:42:06.689950 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:41009
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:41009,127.25.240.125:43119,127.25.240.124:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:06.769454 16285 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:42:06.769680 16285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:06.769697 16285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:06.769717 16285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:06.770879 16285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:06.770919 16285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:06.770933 16285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:06.770946 16285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:06.770957 16285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:06.772397 16285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:41009,127.25.240.125:43119,127.25.240.124:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:41009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:06.772651 16285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:06.772871 16285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 13:42:06.778113 16285 server_base.cc:1034] running on GCE node
W20260924 13:42:06.778075 16291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:06.778087 16293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:06.778083 16290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:06.778566 16285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:06.778863 16285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:06.780017 16285 hybrid_clock.cc:648] HybridClock initialized: now 1790257326779980 us; error 38 us; skew 500 ppm
Sep 24 13:42:06 dist-test-slave-m5z1 krb5kdc[16272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:06.782614 16285 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:42:06.783123 16285 webserver.cc:460] Webserver started at http://127.25.240.126:46195/ using document root <none> and password file <none>
I20260924 13:42:06.783342 16285 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:06.783408 16285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:06.783558 16285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:06.784621 16285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "dc89c14223c14893988d64282fa33d53"
format_stamp: "Formatted at 2026-09-24 13:42:06 on dist-test-slave-m5z1"
server_key: "6d5914fb7b87c7d2344d153aa708f4e3"
server_key_iv: "71448bc533384ce5440314a563fdd2f7"
server_key_version: "encryptionkey@0"
I20260924 13:42:06.784974 16285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "dc89c14223c14893988d64282fa33d53"
format_stamp: "Formatted at 2026-09-24 13:42:06 on dist-test-slave-m5z1"
server_key: "6d5914fb7b87c7d2344d153aa708f4e3"
server_key_iv: "71448bc533384ce5440314a563fdd2f7"
server_key_version: "encryptionkey@0"
I20260924 13:42:06.786355 16285 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:06.787158 16300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:06.787328 16285 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:06.787423 16285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "dc89c14223c14893988d64282fa33d53"
format_stamp: "Formatted at 2026-09-24 13:42:06 on dist-test-slave-m5z1"
server_key: "6d5914fb7b87c7d2344d153aa708f4e3"
server_key_iv: "71448bc533384ce5440314a563fdd2f7"
server_key_version: "encryptionkey@0"
I20260924 13:42:06.787503 16285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:06.802388 16285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:06.804114 16285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:06.804278 16285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:06.808202 16285 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:41009
I20260924 13:42:06.808236 16352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:41009 every 8 connection(s)
I20260924 13:42:06.808594 16285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:06.809432 16353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:06.811053 16353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:06.814436 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 16285
I20260924 13:42:06.814550 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:42:06.814724 26561 external_mini_cluster.cc:1428] Setting key 47733ed151adedf81e673f108d22dec9
Sep 24 13:42:06 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:42:06.820788 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.811665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:41009 (local address 127.0.0.1:35626)
0924 13:42:06.811889 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.811932 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:06.812431 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:06.812731 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:06.812734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:06.813001 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:06.813184 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:06.813189 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.814327 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.814328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:06.815106 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:06.815109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.815596 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.815597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:06.815661 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:06.816195 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:06.816203 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:06.817544 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:06.819325 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:06.819332 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:06.819334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:06.819532 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:06.819781 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:06.819782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:06.819782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:06.819871 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:06.820212 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:06.820217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:06.820277 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 13:42:06.820385 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 13:42:06.820619 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":97,"thread_start_us":108,"threads_started":1}
I20260924 13:42:06.820788 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.811759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35626 (local address 127.25.240.126:41009)
0924 13:42:06.811922 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:06.811932 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:06.812038 (+ 106us) server_negotiation.cc:407] Connection header received
0924 13:42:06.812595 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:06.812597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:06.812629 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:06.812727 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:06.813292 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.814214 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.815203 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.815502 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.815621 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:06.817689 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:06.817696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:06.817696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:06.817703 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:06.819228 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:06.819627 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:06.819630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:06.819630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:06.819710 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:06.819946 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:06.819947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:06.819947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:06.820141 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:06.820232 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:06.820418 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 13:42:06.820556 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.25.240.125@KRBTEST.COM; defaulting to no policy
I20260924 13:42:06.821741 16353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:06.822075 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.821975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49032)
0924 13:42:06.822008 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.822053 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 13:42:06.822234 16301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
I20260924 13:42:06.822614 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.822532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49042)
0924 13:42:06.822576 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.822599 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 13:42:06.822726 16353 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:43119: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab.
Entry for principal kudu/127.25.240.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab.
I20260924 13:42:06.840423 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.125:43119
--webserver_interface=127.25.240.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab
--principal=kudu/127.25.240.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:41009,127.25.240.125:43119,127.25.240.124:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:42:06.871989 16353 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } attempt: 1
I20260924 13:42:06.872530 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.872365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49052)
0924 13:42:06.872455 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.872497 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 13:42:06.872915 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.872819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49060)
0924 13:42:06.872864 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.872892 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 13:42:06.873021 16353 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:43119: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
W20260924 13:42:06.921190 16360 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:42:06.921386 16360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:06.921403 16360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:06.921423 16360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:06.922613 16360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:06.922654 16360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:06.922668 16360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:06.922681 16360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:06.922693 16360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:06.924125 16360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:41009,127.25.240.125:43119,127.25.240.124:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab
--principal=kudu/127.25.240.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.125:43119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.240.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:06.924357 16360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:06.924605 16360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:06.929165 16366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:06.929186 16365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:06.929193 16368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:06.929518 16360 server_base.cc:1034] running on GCE node
I20260924 13:42:06.929786 16360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:06.930058 16360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:06.931224 16360 hybrid_clock.cc:648] HybridClock initialized: now 1790257326931218 us; error 35 us; skew 500 ppm
Sep 24 13:42:06 dist-test-slave-m5z1 krb5kdc[16272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:06.933713 16360 init.cc:377] Logged in from keytab as kudu/127.25.240.125@KRBTEST.COM (short username kudu)
I20260924 13:42:06.934130 16360 webserver.cc:460] Webserver started at http://127.25.240.125:43637/ using document root <none> and password file <none>
I20260924 13:42:06.934367 16360 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:06.934434 16360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:06.934574 16360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:06.935556 16360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/data/instance:
uuid: "bb749710c0784f5ca09931f53b0232f6"
format_stamp: "Formatted at 2026-09-24 13:42:06 on dist-test-slave-m5z1"
server_key: "edb20ce8a208c7505e1206b3f2095662"
server_key_iv: "7ad72d904b5bb8e87358a672a989fc69"
server_key_version: "encryptionkey@0"
I20260924 13:42:06.935840 16360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/wal/instance:
uuid: "bb749710c0784f5ca09931f53b0232f6"
format_stamp: "Formatted at 2026-09-24 13:42:06 on dist-test-slave-m5z1"
server_key: "edb20ce8a208c7505e1206b3f2095662"
server_key_iv: "7ad72d904b5bb8e87358a672a989fc69"
server_key_version: "encryptionkey@0"
I20260924 13:42:06.937054 16360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:06.937744 16375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:06.938014 16360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:06.938156 16360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/wal
uuid: "bb749710c0784f5ca09931f53b0232f6"
format_stamp: "Formatted at 2026-09-24 13:42:06 on dist-test-slave-m5z1"
server_key: "edb20ce8a208c7505e1206b3f2095662"
server_key_iv: "7ad72d904b5bb8e87358a672a989fc69"
server_key_version: "encryptionkey@0"
I20260924 13:42:06.938230 16360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:06.941200 16353 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } attempt: 2
I20260924 13:42:06.941690 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.941514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49076)
0924 13:42:06.941619 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.941659 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 13:42:06.942061 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.941963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49090)
0924 13:42:06.942018 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.942044 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 13:42:06.942170 16353 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:43119: Network error: Client connection negotiation failed: client connection to 127.25.240.125:43119: connect: Connection refused (error 111)
I20260924 13:42:06.955963 16360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:06.958160 16360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:06.958313 16360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:06.962568 16360 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.125:43119
I20260924 13:42:06.962612 16427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.125:43119 every 8 connection(s)
I20260924 13:42:06.962955 16360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
I20260924 13:42:06.963764 16428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:06.964993 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 16360
I20260924 13:42:06.965106 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-1/wal/instance
I20260924 13:42:06.965286 26561 external_mini_cluster.cc:1428] Setting key c79826c28822ed7a74382c99d8237c48
I20260924 13:42:06.965427 16428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:06 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.240.124@KRBTEST.COM; defaulting to no policy
I20260924 13:42:06.974828 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.966219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35628 (local address 127.25.240.126:41009)
0924 13:42:06.966264 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:06.966266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:06.966483 (+ 217us) server_negotiation.cc:407] Connection header received
0924 13:42:06.967230 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:06.967231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:06.967244 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:06.967297 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:06.968109 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.968599 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.969595 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.969810 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.969899 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:06.971992 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:06.971993 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:06.971994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:06.972001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:06.973439 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:06.973883 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:06.973885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:06.973885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:06.973947 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:06.974262 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:06.974264 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:06.974264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:06.974403 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:06.974459 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:06.974698 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 13:42:06.974764 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:06.975066 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.966190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:41009 (local address 127.0.0.1:35628)
0924 13:42:06.966360 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.966394 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:06.967111 (+ 717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:06.967371 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:06.967376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:06.967753 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:06.968014 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:06.968019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.968698 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.968699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:06.969488 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:06.969490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.969909 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.969910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:06.970045 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:06.970559 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:06.970567 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:06.971882 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:06.973537 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:06.973544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:06.973546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:06.973788 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:06.974017 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:06.974018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:06.974019 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:06.974159 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:06.974463 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:06.974468 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:06.974588 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 13:42:06.974667 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:42:06.974778 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.124@KRBTEST.COM" created.
I20260924 13:42:06.975705 16428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:06 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:42:06.983284 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.975996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49100)
0924 13:42:06.976044 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.976056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:06.976095 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:06.976370 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:06.976371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:06.976489 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:06.976532 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:06.976533 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.977448 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.977449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:06.978042 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:06.978044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.978710 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.978711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:06.978768 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:06.979165 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:06.979184 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:06.980280 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:06.982321 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:06.982322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:06.982322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:06.982489 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:06.982701 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:06.982702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:06.982702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:06.982751 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:06.983071 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:06.983072 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:06.983104 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 13:42:06.983152 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 13:42:06.983223 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 13:42:06.983425 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.976056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.25.240.125:43119)
0924 13:42:06.976219 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:06.976224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:06.976244 (+ 20us) server_negotiation.cc:407] Connection header received
0924 13:42:06.976267 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:06.976269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:06.976283 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:06.976362 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:06.976602 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.977351 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.978125 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:06.978629 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:06.978707 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:06.980360 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:06.980368 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:06.980369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:06.980376 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:06.982231 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:06.982573 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:06.982576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:06.982577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:06.982636 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:06.982815 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:06.982816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:06.982817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:06.982993 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:06.983092 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:06.983193 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 13:42:06.983270 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260924 13:42:06.984117 16428 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:06.984474 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.984360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52550)
0924 13:42:06.984413 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.984457 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 13:42:06.984606 16376 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
I20260924 13:42:06.985072 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:06.984975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52560)
0924 13:42:06.985026 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:06.985051 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 13:42:06.985179 16428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:38777: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab.
Entry for principal kudu/127.25.240.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab.
I20260924 13:42:06.993141 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.124:38777
--webserver_interface=127.25.240.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab
--principal=kudu/127.25.240.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:41009,127.25.240.125:43119,127.25.240.124:38777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:42:07.033345 16353 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } attempt: 3
I20260924 13:42:07.034399 16428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } attempt: 1
I20260924 13:42:07.034792 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.034668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52568)
0924 13:42:07.034724 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.034756 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 13:42:07.035174 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.035081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52572)
0924 13:42:07.035124 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.035148 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:42:07.035352 16428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:38777: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:42:07.040860 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.033851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.25.240.125:43119)
0924 13:42:07.033919 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.033922 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.033933 (+ 11us) server_negotiation.cc:407] Connection header received
0924 13:42:07.034006 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.034007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.034018 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.034092 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.034459 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.035238 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.036007 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.036380 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.036458 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.038476 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.038477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.038478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.038483 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.039740 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.040106 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.040107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.040108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.040157 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.040346 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.040346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.040346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.040502 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.040556 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.040702 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.040807 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:42:07.040871 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.033784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49112)
0924 13:42:07.033842 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.033862 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:07.033936 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:07.034098 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:07.034098 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:07.034264 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:07.034353 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.034355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.035343 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.035344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.035930 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.035932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.036459 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.036460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.036511 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.036959 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:07.036963 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:07.038397 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:07.039814 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.039815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.039816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.040032 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.040219 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.040220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.040220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.040288 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.040565 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:07.040566 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:07.040612 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:42:07.040688 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 13:42:07.040779 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 13:42:07.041342 16353 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:07.041688 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.041605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52576)
0924 13:42:07.041642 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.041669 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 13:42:07.041960 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.041898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52580)
0924 13:42:07.041927 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.041948 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 13:42:07.042055 16353 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:38777: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
I20260924 13:42:07.073212 16353 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } attempt: 1
I20260924 13:42:07.073714 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.073591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52586)
0924 13:42:07.073640 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.073682 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 13:42:07.074038 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.073943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52598)
0924 13:42:07.073989 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.074014 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 13:42:07.074126 16353 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:38777: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
W20260924 13:42:07.075166 16435 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:42:07.075363 16435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:07.075413 16435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:07.075469 16435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:07.076637 16435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:07.076678 16435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:07.076692 16435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:07.076745 16435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:07.076766 16435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:07.078305 16435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:41009,127.25.240.125:43119,127.25.240.124:38777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab
--principal=kudu/127.25.240.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.124:38777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.240.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:07.078563 16435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:07.078788 16435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:07.083570 16443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:07.083577 16441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:07.083570 16440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:07.083937 16435 server_base.cc:1034] running on GCE node
I20260924 13:42:07.084189 16435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:07.084483 16435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:07.085636 16435 hybrid_clock.cc:648] HybridClock initialized: now 1790257327085626 us; error 43 us; skew 500 ppm
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257327, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:07.088160 16435 init.cc:377] Logged in from keytab as kudu/127.25.240.124@KRBTEST.COM (short username kudu)
I20260924 13:42:07.088614 16435 webserver.cc:460] Webserver started at http://127.25.240.124:38331/ using document root <none> and password file <none>
I20260924 13:42:07.088805 16435 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:07.088846 16435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:07.088934 16435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:07.090018 16435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/data/instance:
uuid: "73c8ac43cac8420cb9c67c1a5eac61f4"
format_stamp: "Formatted at 2026-09-24 13:42:07 on dist-test-slave-m5z1"
server_key: "b10f784a34cca5294d311f9e48a3cde5"
server_key_iv: "9d28e5f2b06fd9b405c3f2312ef7c2bd"
server_key_version: "encryptionkey@0"
I20260924 13:42:07.090422 16435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/wal/instance:
uuid: "73c8ac43cac8420cb9c67c1a5eac61f4"
format_stamp: "Formatted at 2026-09-24 13:42:07 on dist-test-slave-m5z1"
server_key: "b10f784a34cca5294d311f9e48a3cde5"
server_key_iv: "9d28e5f2b06fd9b405c3f2312ef7c2bd"
server_key_version: "encryptionkey@0"
I20260924 13:42:07.091737 16435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 13:42:07.092415 16450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:07.092605 16435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:07.092693 16435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/wal
uuid: "73c8ac43cac8420cb9c67c1a5eac61f4"
format_stamp: "Formatted at 2026-09-24 13:42:07 on dist-test-slave-m5z1"
server_key: "b10f784a34cca5294d311f9e48a3cde5"
server_key_iv: "9d28e5f2b06fd9b405c3f2312ef7c2bd"
server_key_version: "encryptionkey@0"
I20260924 13:42:07.092764 16435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:07.102566 16435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:07.103544 16428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } attempt: 2
I20260924 13:42:07.103945 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.103810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52606)
0924 13:42:07.103878 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.103916 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 13:42:07.104363 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.104231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52614)
0924 13:42:07.104279 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.104307 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 13:42:07.104478 16428 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:38777: Network error: Client connection negotiation failed: client connection to 127.25.240.124:38777: connect: Connection refused (error 111)
I20260924 13:42:07.104983 16435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:07.105093 16435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:07.108932 16435 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.124:38777
I20260924 13:42:07.108976 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.124:38777 every 8 connection(s)
I20260924 13:42:07.109313 16435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
I20260924 13:42:07.110245 16503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:07.111725 16503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257327, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:42:07.117765 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 16435
I20260924 13:42:07.117870 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790257062855231-26561-0/minicluster-data/master-2/wal/instance
I20260924 13:42:07.118075 26561 external_mini_cluster.cc:1428] Setting key 9b2552601ee68f03671b35b46289e7cf
I20260924 13:42:07.120282 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.112407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.25.240.126:41009)
0924 13:42:07.112479 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.112481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.112663 (+ 182us) server_negotiation.cc:407] Connection header received
0924 13:42:07.113153 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.113154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.113164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.113217 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.113891 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.114352 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.115294 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.115473 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.115531 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.117400 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.117401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.117402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.117408 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.118951 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.119353 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.119354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.119355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.119417 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.119659 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.119661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.119661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.119778 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.119848 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.120113 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.120191 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:42:07.120473 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.112364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:41009 (local address 127.0.0.1:35642)
0924 13:42:07.112548 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.112585 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:07.113071 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:07.113283 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:07.113286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:07.113533 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:07.113815 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.113820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.114446 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.114447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.115221 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.115223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.115548 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.115549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.115644 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.116047 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:07.116055 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:07.117318 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:07.119048 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.119054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.119057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.119274 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.119487 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.119488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.119488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.119589 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.119851 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:07.119858 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:07.119979 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 13:42:07.120091 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 13:42:07.120245 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260924 13:42:07.121016 16503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257327, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:42:07.127075 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.118459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.25.240.124:38777)
0924 13:42:07.118655 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.118659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.118687 (+ 28us) server_negotiation.cc:407] Connection header received
0924 13:42:07.118755 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.118757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.118770 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.118834 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.119433 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.120104 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.121128 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.121533 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.121657 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.124012 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.124018 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.124019 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.124038 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.125745 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.126175 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.126179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.126179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.126242 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.126426 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.126427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.126427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.126612 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.126771 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.126853 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.126960 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":80,"threads_started":1}
I20260924 13:42:07.127893 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.121255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:43119 (local address 127.0.0.1:49114)
0924 13:42:07.121304 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.121314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:07.121353 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:07.121489 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:07.121489 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:07.121660 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:07.121710 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.121713 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.122488 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.122489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.123067 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.123069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.123402 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.123403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.123482 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.123792 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:07.123797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:07.124931 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:07.126719 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.126720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.126720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.126967 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.127283 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.127284 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.127284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.127333 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.127601 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:07.127602 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:07.127641 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 13:42:07.127719 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 13:42:07.127810 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 13:42:07.127921 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.121343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.25.240.125:43119)
0924 13:42:07.121389 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.121392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.121398 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:42:07.121420 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.121421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.121430 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.121484 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.121802 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.122391 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.123147 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.123330 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.123382 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.125026 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.125028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.125028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.125035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.126650 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.127048 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.127049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.127050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.127108 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.127408 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.127409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.127410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.127527 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.127597 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.127759 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.127834 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:07.128250 16503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257327, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:42:07.134711 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.128519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.25.240.124:38777)
0924 13:42:07.128562 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.128564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.128573 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:07.128615 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.128616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.128625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.128689 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.128936 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.129428 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.130121 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.130325 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.130390 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.132270 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.132271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.132271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.132277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.133688 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.134079 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.134080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.134080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.134127 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.134318 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.134319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.134319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.134403 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.134445 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.134594 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.134658 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:07.134833 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.128475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52626)
0924 13:42:07.128520 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.128530 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:07.128566 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:07.128691 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:07.128691 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:07.128840 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:07.128868 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.128869 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.129508 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.129509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.130046 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.130048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.130404 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.130405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.130440 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.130919 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:07.130923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:07.132183 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:07.133764 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.133765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.133766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.133995 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.134191 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.134192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.134193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.134257 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.134452 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:07.134453 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:07.134500 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 13:42:07.134579 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 13:42:07.134719 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":48}
I20260924 13:42:07.135952 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.127864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.25.240.126:41009)
0924 13:42:07.127900 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.127902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.127910 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:07.128046 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.128048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.128059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.128094 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.128738 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.129278 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.130018 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.130270 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.130319 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.133006 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.133008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.133008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.133015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.134725 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.135130 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.135132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.135132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.135220 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.135410 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.135411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.135411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.135567 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.135755 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.135793 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.135867 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:07.136565 16503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4: Bootstrap starting.
I20260924 13:42:07.137682 16503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:07.137987 16503 log.cc:824] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:07.138801 16503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4: No bootstrap required, opened a new log
I20260924 13:42:07.140621 16503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } }
I20260924 13:42:07.140743 16503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8ac43cac8420cb9c67c1a5eac61f4, State: Initialized, Role: FOLLOWER
I20260924 13:42:07.140830 16503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } }
I20260924 13:42:07.141099 16509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.141161 16503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:07.141229 16509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
W20260924 13:42:07.143210 16520 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:07.143304 16520 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:07.145416 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.137013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.25.240.125:43119)
0924 13:42:07.137088 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.137090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.137100 (+ 10us) server_negotiation.cc:407] Connection header received
0924 13:42:07.137127 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.137128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.137141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.137221 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.137684 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.138338 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.139120 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.139393 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.139467 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.142299 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.142301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.142301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.142309 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.144096 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.144526 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.144527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.144528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.144592 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.144855 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.144856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.144856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.145004 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.145199 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.145243 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.145322 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 13:42:07.143532 16435 master_runner.cc:186] Error getting master registration for 127.25.240.126:41009: OK, instance_id { permanent_uuid: "dc89c14223c14893988d64282fa33d53" instance_seqno: 1790257326807080 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:42:07.146054 16435 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:42:07.146994 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:07.147915 16327 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "dc89c14223c14893988d64282fa33d53"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35646
I20260924 13:42:07.149258 16353 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } attempt: 2
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
I20260924 13:42:07.157438 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.149617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52630 (local address 127.25.240.124:38777)
0924 13:42:07.149673 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.149675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.149684 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:07.149704 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.149705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.149714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.149765 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.150179 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.151020 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.152025 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.152357 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.152440 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.154787 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.154789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.154789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.154796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.156221 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.156639 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.156640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.156640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.156687 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.156897 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.156898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.156898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.157037 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.157115 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.157268 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.157344 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:07.157438 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.149518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52630)
0924 13:42:07.149567 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.149607 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:07.149648 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:07.149799 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:07.149801 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:07.150007 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:07.150068 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.150070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.151119 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.151120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.151939 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.151941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.152445 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.152446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.152503 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.152976 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:07.152982 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:07.154678 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:07.156297 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.156298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.156298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.156556 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.156759 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.156760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.156760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.156824 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.157107 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:07.157108 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:07.157154 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 13:42:07.157243 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 13:42:07.157343 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 13:42:07.158984 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53: Bootstrap starting.
I20260924 13:42:07.159626 16353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:07.159947 16353 log.cc:824] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:07.160748 16353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53: No bootstrap required, opened a new log
I20260924 13:42:07.162170 16353 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } }
I20260924 13:42:07.162312 16353 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc89c14223c14893988d64282fa33d53, State: Initialized, Role: FOLLOWER
I20260924 13:42:07.162415 16353 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } }
I20260924 13:42:07.162750 16327 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 13:42:07.162729 16353 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:07.162825 16327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:07.162716 16523 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.162946 16523 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:07.163699 16327 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } }
I20260924 13:42:07.164637 16477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc89c14223c14893988d64282fa33d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4"
I20260924 13:42:07.164775 16477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [term 0 FOLLOWER]: Advancing to term 1
W20260924 13:42:07.165131 16534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:07.165236 16534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:07.165781 16477 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc89c14223c14893988d64282fa33d53 in term 1.
I20260924 13:42:07.166040 16301 leader_election.cc:302] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c8ac43cac8420cb9c67c1a5eac61f4, dc89c14223c14893988d64282fa33d53; no voters:
I20260924 13:42:07.166162 16523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:07.166304 16523 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [term 1 LEADER]: Becoming Leader. State: Replica: dc89c14223c14893988d64282fa33d53, State: Running, Role: LEADER
I20260924 13:42:07.166388 16523 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } }
I20260924 13:42:07.166862 16327 leader_election.cc:288] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [CANDIDATE]: Term 1 election: Requested vote from peers bb749710c0784f5ca09931f53b0232f6 (127.25.240.125:43119), 73c8ac43cac8420cb9c67c1a5eac61f4 (127.25.240.124:38777)
I20260924 13:42:07.166828 16536 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc89c14223c14893988d64282fa33d53. Latest consensus state: current_term: 1 leader_uuid: "dc89c14223c14893988d64282fa33d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.167217 16536 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:07.165334 16285 master_runner.cc:186] Error getting master registration for 127.25.240.125:43119: OK, instance_id { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" instance_seqno: 1790257326961221 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:42:07.167863 16285 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:42:07.167999 16538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:07.168215 16538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:07.170017 16477 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [term 1 FOLLOWER]: Refusing update from remote peer dc89c14223c14893988d64282fa33d53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:07.170439 16536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:07.171805 16509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc89c14223c14893988d64282fa33d53. Latest consensus state: current_term: 1 leader_uuid: "dc89c14223c14893988d64282fa33d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.171892 16509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:07.173813 16536 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc89c14223c14893988d64282fa33d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.173893 16536 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:07.174420 16509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc89c14223c14893988d64282fa33d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.174504 16509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:07.174945 16543 mvcc.cc:205] Tried to move back new op lower bound from 7332894012085432320 to 7332894012074192896. Current Snapshot: MvccSnapshot[applied={T|T < 7332894012085432320}]
I20260924 13:42:07.175923 16547 mvcc.cc:205] Tried to move back new op lower bound from 7332894012085432320 to 7332894012074192896. Current Snapshot: MvccSnapshot[applied={T|T < 7332894012085432320}]
I20260924 13:42:07.176838 16538 catalog_manager.cc:1347] Generated new cluster ID: 6b175d65e001483c8c3b0f712bbe13a1
I20260924 13:42:07.176885 16538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:07.178922 16432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.169186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.25.240.125:43119)
0924 13:42:07.169237 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.169239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.169996 (+ 757us) server_negotiation.cc:407] Connection header received
0924 13:42:07.170109 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.170111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.170123 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.170156 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.171060 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.171910 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.173190 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.173526 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.173592 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.175569 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.175571 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.175571 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.175579 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.177471 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.177950 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.177952 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.177952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.178014 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.178234 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.178235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.178235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.178371 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.178467 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.178706 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.178801 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 13:42:07.190847 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.169405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.25.240.124:38777)
0924 13:42:07.169448 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.169450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.181704 (+ 12254us) server_negotiation.cc:407] Connection header received
0924 13:42:07.181808 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.181809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.181820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.181850 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.182657 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.183465 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.184700 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.185054 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.185104 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.187637 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.187639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.187639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.187647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.189451 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.189950 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.189951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.189952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.190012 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.190230 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.190231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.190231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.190369 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.190456 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.190658 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.190741 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 13:42:07.192399 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.169653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35652 (local address 127.25.240.126:41009)
0924 13:42:07.169695 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.169697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.179356 (+ 9659us) server_negotiation.cc:407] Connection header received
0924 13:42:07.179481 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.179482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.179494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.179526 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.180266 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.181080 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.182347 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.182706 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.182751 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.184424 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.184426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.184427 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.184434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.186516 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.191498 (+ 4982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.191499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.191500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.191566 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.191773 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.191774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.191774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.191915 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.192000 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.192211 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.192299 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:07.195645 16428 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } attempt: 3
Sep 24 13:42:07 dist-test-slave-m5z1 krb5kdc[16272](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257326, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
I20260924 13:42:07.204982 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.195951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:38777 (local address 127.0.0.1:52638)
0924 13:42:07.196044 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:07.196056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:07.196096 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:07.196416 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:07.196417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:07.196599 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:07.196651 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.196653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.197755 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.197756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.198733 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:07.198736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.199458 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.199459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:07.199521 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.200048 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:07.200054 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:07.201716 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:07.203800 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.203802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.203802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.204046 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.204264 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:07.204265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:07.204265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:07.204325 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:07.204625 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:07.204626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:07.204676 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 13:42:07.204744 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 13:42:07.204851 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 13:42:07.205374 16507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:07.196188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.25.240.124:38777)
0924 13:42:07.196263 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.196265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.196272 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:42:07.196293 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.196294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.196304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.196365 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.196736 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.197628 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.199003 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:07.199330 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:07.199398 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:07.201832 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:07.201834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:07.201835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:07.201842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:07.203696 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.204133 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.204134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.204135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.204193 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:07.204395 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:07.204396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:07.204396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:07.204534 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:07.204597 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:07.205192 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 13:42:07.205276 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 13:42:07.207248 16428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6: Bootstrap starting.
I20260924 13:42:07.208088 16428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:07.208418 16428 log.cc:824] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:07.210260 16428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6: No bootstrap required, opened a new log
I20260924 13:42:07.212435 16428 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } }
I20260924 13:42:07.212599 16428 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb749710c0784f5ca09931f53b0232f6, State: Initialized, Role: FOLLOWER
I20260924 13:42:07.212688 16428 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } }
I20260924 13:42:07.213057 16428 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:07.213105 16402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dc89c14223c14893988d64282fa33d53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bb749710c0784f5ca09931f53b0232f6"
I20260924 13:42:07.213228 16402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:07.214098 16402 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc89c14223c14893988d64282fa33d53 in term 1.
I20260924 13:42:07.214295 16401 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [term 1 FOLLOWER]: Refusing update from remote peer dc89c14223c14893988d64282fa33d53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:07.214730 16523 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 13:42:07.215231 16563 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:07.215333 16563 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:07.217882 16565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc89c14223c14893988d64282fa33d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.218065 16565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:07.218168 16565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc89c14223c14893988d64282fa33d53. Latest consensus state: current_term: 1 leader_uuid: "dc89c14223c14893988d64282fa33d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.218237 16565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:07.218271 16565 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dc89c14223c14893988d64282fa33d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.218315 16565 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:07.232389 16523 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dc89c14223c14893988d64282fa33d53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc89c14223c14893988d64282fa33d53" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 41009 } } peers { permanent_uuid: "bb749710c0784f5ca09931f53b0232f6" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 43119 } } peers { permanent_uuid: "73c8ac43cac8420cb9c67c1a5eac61f4" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 38777 } } }
I20260924 13:42:07.232534 16523 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53 [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:07.234767 16538 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:07.235591 16538 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:07.245841 16538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc89c14223c14893988d64282fa33d53: Generated new TSK 0
I20260924 13:42:07.246055 16538 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:42:07.278683 16356 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.276904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.25.240.126:41009)
0924 13:42:07.276955 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.276958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.277969 (+ 1011us) server_negotiation.cc:407] Connection header received
0924 13:42:07.278080 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.278081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.278094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.278126 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.278606 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35656: BlockingRecv error: recv got EOF from 127.0.0.1:35656 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 13:42:07.279465 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.277326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49148 (local address 127.25.240.125:43119)
0924 13:42:07.277372 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.277374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.278938 (+ 1564us) server_negotiation.cc:407] Connection header received
0924 13:42:07.279037 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.279038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.279049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.279080 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.279400 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49148: BlockingRecv error: recv got EOF from 127.0.0.1:49148 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 13:42:07.280424 16507 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.277697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52644 (local address 127.25.240.124:38777)
0924 13:42:07.277740 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.277742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.279705 (+ 1963us) server_negotiation.cc:407] Connection header received
0924 13:42:07.279801 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.279802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.279814 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.279843 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.280368 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52644: BlockingRecv error: recv got EOF from 127.0.0.1:52644 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 13:42:07.282909 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.282076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.25.240.125:43119)
0924 13:42:07.282109 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.282111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.282361 (+ 250us) server_negotiation.cc:407] Connection header received
0924 13:42:07.282469 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.282470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.282487 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.282516 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.282851 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49158: BlockingRecv error: recv got EOF from 127.0.0.1:49158 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 13:42:07.285928 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.285142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.25.240.125:43119)
0924 13:42:07.285174 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.285176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.285390 (+ 214us) server_negotiation.cc:407] Connection header received
0924 13:42:07.285484 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.285485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.285494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.285535 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.285886 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49174: BlockingRecv error: recv got EOF from 127.0.0.1:49174 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 13:42:07.289899 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.289081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.25.240.125:43119)
0924 13:42:07.289116 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.289118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.289324 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:42:07.289454 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.289454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.289465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.289493 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.289858 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49182: BlockingRecv error: recv got EOF from 127.0.0.1:49182 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:42:07.296013 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.295115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49184 (local address 127.25.240.125:43119)
0924 13:42:07.295151 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.295153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.295436 (+ 283us) server_negotiation.cc:407] Connection header received
0924 13:42:07.295547 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.295548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.295558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.295603 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.295969 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49184: BlockingRecv error: recv got EOF from 127.0.0.1:49184 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:42:07.306247 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.305351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.25.240.125:43119)
0924 13:42:07.305397 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.305399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.305668 (+ 269us) server_negotiation.cc:407] Connection header received
0924 13:42:07.305769 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.305771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.305782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.305814 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.306191 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49186: BlockingRecv error: recv got EOF from 127.0.0.1:49186 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 13:42:07.324558 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.323644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.25.240.125:43119)
0924 13:42:07.323689 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.323692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.323945 (+ 253us) server_negotiation.cc:407] Connection header received
0924 13:42:07.324049 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.324050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.324063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.324093 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.324485 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49194: BlockingRecv error: recv got EOF from 127.0.0.1:49194 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 13:42:07.358675 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.357921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.25.240.125:43119)
0924 13:42:07.357972 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.357975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.358108 (+ 133us) server_negotiation.cc:407] Connection header received
0924 13:42:07.358191 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.358192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.358203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.358255 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.358605 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49208: BlockingRecv error: recv got EOF from 127.0.0.1:49208 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 13:42:07.424666 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.423976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.25.240.125:43119)
0924 13:42:07.424078 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.424080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.424094 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:42:07.424163 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.424165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.424175 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.424234 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.424609 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49220: BlockingRecv error: recv got EOF from 127.0.0.1:49220 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 13:42:07.554653 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.553915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.25.240.125:43119)
0924 13:42:07.553993 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.553996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.554098 (+ 102us) server_negotiation.cc:407] Connection header received
0924 13:42:07.554146 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.554147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.554157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.554230 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.554580 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49226: BlockingRecv error: recv got EOF from 127.0.0.1:49226 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260924 13:42:07.812342 16432 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:07.811777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.25.240.125:43119)
0924 13:42:07.811848 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:07.811850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:07.811876 (+ 26us) server_negotiation.cc:407] Connection header received
0924 13:42:07.811936 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:07.811937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:07.811947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:07.811989 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:07.812300 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49234: BlockingRecv error: recv got EOF from 127.0.0.1:49234 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 13:42:08.143766 16520 catalog_manager.cc:1259] Loaded cluster ID: 6b175d65e001483c8c3b0f712bbe13a1
I20260924 13:42:08.143844 16520 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4: loading cluster ID for follower catalog manager: success
I20260924 13:42:08.144601 16520 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4: acquiring CA information for follower catalog manager: success
I20260924 13:42:08.144840 16520 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 73c8ac43cac8420cb9c67c1a5eac61f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:42:08.215807 16563 catalog_manager.cc:1259] Loaded cluster ID: 6b175d65e001483c8c3b0f712bbe13a1
I20260924 13:42:08.215911 16563 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6: loading cluster ID for follower catalog manager: success
I20260924 13:42:08.216743 16563 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6: acquiring CA information for follower catalog manager: success
I20260924 13:42:08.217020 16563 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb749710c0784f5ca09931f53b0232f6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:42:08.328132 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:08.325527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.25.240.125:43119)
0924 13:42:08.325645 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:08.325647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:08.325733 (+ 86us) server_negotiation.cc:407] Connection header received
0924 13:42:08.325768 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:08.325769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:08.325817 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:08.325939 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:08.326083 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.326589 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.327337 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.327540 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.327596 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:08.327741 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:08.327854 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:08.327917 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:08.327987 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 13:42:08.328004 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260924 13:42:08.332346 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:08.330113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.25.240.124:38777)
0924 13:42:08.330254 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:08.330255 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:08.330263 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:08.330326 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:08.330326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:08.330364 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:08.330433 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:08.330533 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.331018 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.331683 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.331866 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.331939 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:08.332040 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:08.332127 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:08.332179 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:08.332222 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 13:42:08.332238 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260924 13:42:08.336912 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:08.334233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.25.240.125:43119)
0924 13:42:08.334274 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:08.334276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:08.334407 (+ 131us) server_negotiation.cc:407] Connection header received
0924 13:42:08.334467 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:08.334468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:08.334480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:08.334524 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:08.334702 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.335273 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.336201 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.336391 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.336478 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:08.336631 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:08.336663 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:08.336744 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:08.336806 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 13:42:08.336826 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:08.337373 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:08.334484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.25.240.124:38777)
0924 13:42:08.334518 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:08.334520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:08.334623 (+ 103us) server_negotiation.cc:407] Connection header received
0924 13:42:08.334753 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:08.334754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:08.334765 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:08.334813 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:08.334961 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.335620 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.336595 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.336887 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.336936 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:08.337116 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:08.337144 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:08.337170 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:08.337286 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 13:42:08.337307 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:08.337381 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:08.334138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35672 (local address 127.25.240.126:41009)
0924 13:42:08.334243 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:08.334289 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:08.334298 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:08.334344 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:08.334345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:08.334420 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:08.334502 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:08.334604 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.335393 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.336465 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:08.336789 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:08.336844 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:08.337070 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:08.337174 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:08.337238 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:08.337288 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 13:42:08.337308 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260924 13:42:08.338989 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 16285
I20260924 13:42:08.344623 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 16360
I20260924 13:42:08.349370 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 16435
2026-09-24T13:42:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2816 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 13:42:08 dist-test-slave-m5z1 krb5kdc[16652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 13:42:08 dist-test-slave-m5z1 krb5kdc[16652](info): set up 2 sockets
Sep 24 13:42:08 dist-test-slave-m5z1 krb5kdc[16652](info): commencing operation
krb5kdc: starting...
W20260924 13:42:09.406209 26561 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/test-user.keytab.
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T13:42:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T13:42:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.240.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
Entry for principal kudu/127.25.240.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab.
I20260924 13:42:09.528931 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37011
--webserver_interface=127.25.240.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:37011,127.25.240.125:38055,127.25.240.124:37839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 13:42:09.608665 16665 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:42:09.608908 16665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:09.608946 16665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:09.608984 16665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:09.610132 16665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:09.610194 16665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:09.610220 16665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:09.610250 16665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:09.610273 16665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:09.611718 16665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:37011,127.25.240.125:38055,127.25.240.124:37839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.126.keytab
--principal=kudu/127.25.240.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.126:37011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.240.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:09.611979 16665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:09.612188 16665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:09.616953 16671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:09.616945 16673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:09.616941 16670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:09.617129 16665 server_base.cc:1034] running on GCE node
I20260924 13:42:09.617336 16665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:09.617664 16665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:09.618830 16665 hybrid_clock.cc:648] HybridClock initialized: now 1790257329618817 us; error 24 us; skew 500 ppm
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:09.621583 16665 init.cc:377] Logged in from keytab as kudu/127.25.240.126@KRBTEST.COM (short username kudu)
I20260924 13:42:09.622099 16665 webserver.cc:460] Webserver started at http://127.25.240.126:37915/ using document root <none> and password file <none>
I20260924 13:42:09.622345 16665 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:09.622411 16665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:09.622542 16665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:09.623610 16665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/data/instance:
uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "e5fd68047e2e1c1b1e00de68fcb484c9"
server_key_iv: "078b75941161522dd9276348dcd40de5"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.623966 16665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/wal/instance:
uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "e5fd68047e2e1c1b1e00de68fcb484c9"
server_key_iv: "078b75941161522dd9276348dcd40de5"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.625252 16665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 13:42:09.626082 16680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:09.626252 16665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 13:42:09.626346 16665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/wal
uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "e5fd68047e2e1c1b1e00de68fcb484c9"
server_key_iv: "078b75941161522dd9276348dcd40de5"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.626422 16665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:09.633890 16665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:09.635816 16665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:09.635965 16665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:09.640375 16665 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.126:37011
I20260924 13:42:09.640436 16732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.126:37011 every 8 connection(s)
I20260924 13:42:09.640765 16665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/data/info.pb
I20260924 13:42:09.641702 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:09.643169 16733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:09.643430 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 16665
I20260924 13:42:09.643514 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-0/wal/instance
I20260924 13:42:09.643738 26561 external_mini_cluster.cc:1428] Setting key cfd7422e54043631342af442d69eaee3
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.240.125@KRBTEST.COM; defaulting to no policy
I20260924 13:42:09.653712 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.643907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37011 (local address 127.0.0.1:37294)
0924 13:42:09.644133 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.644180 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:09.644659 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:09.644949 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:09.644951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:09.645320 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:09.645547 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.645553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.646861 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.646862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.647604 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.647606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.648324 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.648325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.648382 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.648897 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:09.648905 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:09.650487 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:09.652317 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.652323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.652325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.652504 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.652732 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.652733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.652733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.652854 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.653205 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:09.653209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:09.653261 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 13:42:09.653330 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 13:42:09.653446 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260924 13:42:09.653713 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.644171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.25.240.126:37011)
0924 13:42:09.644316 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.644321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.644361 (+ 40us) server_negotiation.cc:407] Connection header received
0924 13:42:09.644813 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.644815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.644838 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.644936 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.645673 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.646767 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.647693 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.648230 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.648359 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.650578 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.650586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.650586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.650593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.652247 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.652582 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.652584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.652585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.652656 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.652920 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.652921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.652921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.653144 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.653208 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.653376 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.653502 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260924 13:42:09.654415 16733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.125@KRBTEST.COM" created.
I20260924 13:42:09.654767 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.654649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37052)
0924 13:42:09.654698 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.654742 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 13:42:09.654855 16681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
I20260924 13:42:09.655273 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.655198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37060)
0924 13:42:09.655232 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.655252 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 13:42:09.655354 16733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:38055: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab.
Entry for principal kudu/127.25.240.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab.
I20260924 13:42:09.669384 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.125:38055
--webserver_interface=127.25.240.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab
--principal=kudu/127.25.240.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:37011,127.25.240.125:38055,127.25.240.124:37839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:42:09.704577 16733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } attempt: 1
I20260924 13:42:09.705070 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.704916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37068)
0924 13:42:09.704979 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.705022 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:42:09.705422 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.705328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37072)
0924 13:42:09.705373 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.705397 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 13:42:09.705545 16733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:38055: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
W20260924 13:42:09.748495 16740 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:42:09.748735 16740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:09.748765 16740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:09.748803 16740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:09.749998 16740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:09.750054 16740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:09.750080 16740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:09.750111 16740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:09.750141 16740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:09.751560 16740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:37011,127.25.240.125:38055,127.25.240.124:37839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.125.keytab
--principal=kudu/127.25.240.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.125:38055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.240.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:09.751827 16740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:09.752055 16740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:09.756758 16748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:09.756839 16745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:09.756767 16746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:09.757026 16740 server_base.cc:1034] running on GCE node
I20260924 13:42:09.757258 16740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:09.757597 16740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:09.758754 16740 hybrid_clock.cc:648] HybridClock initialized: now 1790257329758734 us; error 40 us; skew 500 ppm
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:09.761277 16740 init.cc:377] Logged in from keytab as kudu/127.25.240.125@KRBTEST.COM (short username kudu)
I20260924 13:42:09.761767 16740 webserver.cc:460] Webserver started at http://127.25.240.125:35843/ using document root <none> and password file <none>
I20260924 13:42:09.761972 16740 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:09.762013 16740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:09.762097 16740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:09.763091 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/data/instance:
uuid: "af141c61560746899b968e2b6bac650e"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "a4f3d8c6c88c192408f128c3ddc8b466"
server_key_iv: "41ffe13a60f3581adcf237840e195ea8"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.763381 16740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/wal/instance:
uuid: "af141c61560746899b968e2b6bac650e"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "a4f3d8c6c88c192408f128c3ddc8b466"
server_key_iv: "41ffe13a60f3581adcf237840e195ea8"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.764639 16740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 13:42:09.765350 16755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:09.765528 16740 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 13:42:09.765633 16740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/wal
uuid: "af141c61560746899b968e2b6bac650e"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "a4f3d8c6c88c192408f128c3ddc8b466"
server_key_iv: "41ffe13a60f3581adcf237840e195ea8"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.765690 16740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:09.773707 16733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } attempt: 2
I20260924 13:42:09.774133 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.774008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37084)
0924 13:42:09.774067 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.774104 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 13:42:09.774461 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.774384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37086)
0924 13:42:09.774423 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.774444 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 13:42:09.774580 16733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.125:38055: Network error: Client connection negotiation failed: client connection to 127.25.240.125:38055: connect: Connection refused (error 111)
I20260924 13:42:09.790863 16740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:09.793210 16740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:09.793375 16740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:09.797171 16740 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.125:38055
I20260924 13:42:09.797245 16807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.125:38055 every 8 connection(s)
I20260924 13:42:09.797613 16740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/data/info.pb
I20260924 13:42:09.798561 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:09.800359 16808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:09.804087 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 16740
I20260924 13:42:09.804203 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-1/wal/instance
I20260924 13:42:09.804391 26561 external_mini_cluster.cc:1428] Setting key 8ed9f2ece2a6330e22db02e9f7e29e4c
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:42:09.809687 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.801068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.25.240.126:37011)
0924 13:42:09.801159 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.801161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.801309 (+ 148us) server_negotiation.cc:407] Connection header received
0924 13:42:09.801863 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.801864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.801876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.801933 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.802533 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.803015 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.804036 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.804276 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.804349 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.806615 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.806616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.806617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.806623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.808071 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.808582 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.808583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.808583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.808638 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.808939 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.808940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.808940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.809044 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.809096 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.809477 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.809609 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 13:42:09.809861 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.800935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37011 (local address 127.0.0.1:37298)
0924 13:42:09.801177 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.801225 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:09.801759 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:09.802009 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:09.802012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:09.802242 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:09.802425 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.802428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.803128 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.803129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.803942 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.803945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.804358 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.804359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.804518 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.805037 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:09.805045 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:09.806448 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:09.808157 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.808164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.808167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.808493 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.808711 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.808712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.808712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.808862 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.809112 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:09.809119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:09.809297 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 13:42:09.809426 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 13:42:09.809626 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
I20260924 13:42:09.810469 16808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.240.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.240.124@KRBTEST.COM" created.
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:42:09.818631 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.810734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37088)
0924 13:42:09.810783 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.810794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:09.810837 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:09.811144 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:09.811144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:09.811253 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:09.811299 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.811300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.812580 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.812581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.813297 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.813300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.813892 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.813893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.813952 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.814393 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:09.814398 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:09.815689 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:09.817627 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.817628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.817628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.817786 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.817998 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.817999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.817999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.818048 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.818371 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:09.818372 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:09.818427 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 13:42:09.818497 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 13:42:09.818568 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 13:42:09.818797 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.810810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.25.240.125:38055)
0924 13:42:09.810970 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.810975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.811004 (+ 29us) server_negotiation.cc:407] Connection header received
0924 13:42:09.811028 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.811030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.811045 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.811126 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.811373 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.812492 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.813389 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.813808 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.813868 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.815790 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.815798 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.815798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.815804 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.817515 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.817861 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.817864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.817865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.817930 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.818106 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.818106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.818107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.818282 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.818364 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.818555 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.818643 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260924 13:42:09.819689 16808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:09.820067 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.819950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37566)
0924 13:42:09.820000 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.820049 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 13:42:09.820236 16758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
I20260924 13:42:09.820627 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.820565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37568)
0924 13:42:09.820590 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.820613 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 13:42:09.820753 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:37839: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.240.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab.
Entry for principal kudu/127.25.240.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab.
I20260924 13:42:09.833388 26561 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
/tmp/dist-test-taskFHYdCJ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.124:37839
--webserver_interface=127.25.240.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab
--principal=kudu/127.25.240.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.240.84:33429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.240.126:37011,127.25.240.125:38055,127.25.240.124:37839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 13:42:09.865736 16733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } attempt: 3
I20260924 13:42:09.869983 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } attempt: 1
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:42:09.870374 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.870242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37576)
0924 13:42:09.870301 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.870338 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 13:42:09.870808 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.870687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37592)
0924 13:42:09.870742 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.870775 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 13:42:09.870901 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:37839: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
I20260924 13:42:09.873279 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.866134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37094)
0924 13:42:09.866206 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.866218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:09.866271 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:09.866426 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:09.866427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:09.866592 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:09.866656 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.866657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.867340 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.867342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.867960 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.867966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.868352 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.868353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.868438 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.868993 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:09.869000 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:09.870739 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:09.872391 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.872392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.872393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.872559 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.872755 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.872756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.872756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.872807 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.873054 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:09.873054 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:09.873091 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 13:42:09.873140 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 13:42:09.873210 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 13:42:09.873312 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.866211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37094 (local address 127.25.240.125:38055)
0924 13:42:09.866279 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.866281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.866293 (+ 12us) server_negotiation.cc:407] Connection header received
0924 13:42:09.866325 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.866326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.866339 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.866420 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.866743 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.867253 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.868052 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.868277 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.868354 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.870843 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.870844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.870844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.870851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.872304 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.872630 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.872631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.872631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.872691 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.872864 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.872865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.872865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.872993 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.873059 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.873163 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.873235 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 13:42:09.873750 16733 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:09.874168 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.874071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37606)
0924 13:42:09.874117 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.874143 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 13:42:09.874475 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.874396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37620)
0924 13:42:09.874433 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.874455 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 13:42:09.874565 16733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:37839: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
I20260924 13:42:09.905717 16733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } attempt: 1
I20260924 13:42:09.906234 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.906089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37622)
0924 13:42:09.906162 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.906201 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 13:42:09.906618 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.906540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37624)
0924 13:42:09.906580 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.906603 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 13:42:09.906725 16733 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:37839: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
W20260924 13:42:09.915766 16815 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 13:42:09.915956 16815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 13:42:09.915973 16815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 13:42:09.915993 16815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 13:42:09.917068 16815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 13:42:09.917100 16815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 13:42:09.917114 16815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 13:42:09.917127 16815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 13:42:09.917140 16815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 13:42:09.918632 16815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.240.84:33429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.240.126:37011,127.25.240.125:38055,127.25.240.124:37839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/krb5kdc/kudu_127.25.240.124.keytab
--principal=kudu/127.25.240.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.240.124:37839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.240.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 12:29:37 UTC on ccycloud.quasar-iqzbxn.root.comops.site
build id 4670106
I20260924 13:42:09.918820 16815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 13:42:09.918977 16815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 13:42:09.923846 16823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:09.923889 16821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 13:42:09.923899 16820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 13:42:09.924119 16815 server_base.cc:1034] running on GCE node
I20260924 13:42:09.924404 16815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 13:42:09.924707 16815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 13:42:09.925894 16815 hybrid_clock.cc:648] HybridClock initialized: now 1790257329925864 us; error 39 us; skew 500 ppm
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 13:42:09.928349 16815 init.cc:377] Logged in from keytab as kudu/127.25.240.124@KRBTEST.COM (short username kudu)
I20260924 13:42:09.928843 16815 webserver.cc:460] Webserver started at http://127.25.240.124:44661/ using document root <none> and password file <none>
I20260924 13:42:09.929093 16815 fs_manager.cc:359] Metadata directory not provided
I20260924 13:42:09.929164 16815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 13:42:09.929307 16815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 13:42:09.930464 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/data/instance:
uuid: "99c60959642142bea376fc9279610acb"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "b6f5d8d43a8e5c748de98497247e3053"
server_key_iv: "5c811ce997f8595995f780fd39bf101a"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.930820 16815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/wal/instance:
uuid: "99c60959642142bea376fc9279610acb"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "b6f5d8d43a8e5c748de98497247e3053"
server_key_iv: "5c811ce997f8595995f780fd39bf101a"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.932108 16815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 13:42:09.932771 16830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 13:42:09.932931 16815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 13:42:09.933033 16815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/data,/tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/wal
uuid: "99c60959642142bea376fc9279610acb"
format_stamp: "Formatted at 2026-09-24 13:42:09 on dist-test-slave-m5z1"
server_key: "b6f5d8d43a8e5c748de98497247e3053"
server_key_iv: "5c811ce997f8595995f780fd39bf101a"
server_key_version: "encryptionkey@0"
I20260924 13:42:09.933115 16815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 13:42:09.939033 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } attempt: 2
I20260924 13:42:09.939411 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.939291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37636)
0924 13:42:09.939347 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.939383 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 13:42:09.939786 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.939697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37638)
0924 13:42:09.939740 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.939765 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 13:42:09.939904 16808 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.240.124:37839: Network error: Client connection negotiation failed: client connection to 127.25.240.124:37839: connect: Connection refused (error 111)
I20260924 13:42:09.949071 16815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 13:42:09.951387 16815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 13:42:09.951550 16815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 13:42:09.955492 16815 rpc_server.cc:307] RPC server started. Bound to: 127.25.240.124:37839
I20260924 13:42:09.955523 16882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.240.124:37839 every 8 connection(s)
I20260924 13:42:09.955871 16815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/data/info.pb
I20260924 13:42:09.956735 16883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 13:42:09.958048 26561 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu as pid 16815
I20260924 13:42:09.958164 26561 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFHYdCJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790257062855231-26561-0/minicluster-data/master-2/wal/instance
I20260924 13:42:09.958379 26561 external_mini_cluster.cc:1428] Setting key 9cdff2fe10a4765ea7c3aebd0e541a79
I20260924 13:42:09.958585 16883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
I20260924 13:42:09.967801 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.959251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.25.240.126:37011)
0924 13:42:09.959300 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.959302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.959420 (+ 118us) server_negotiation.cc:407] Connection header received
0924 13:42:09.959734 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.959736 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.959748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.959801 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.960539 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.961275 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.962474 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.962696 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.962765 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.964992 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.964994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.964994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.965001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.966415 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.966857 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.966858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.966858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.966917 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.967175 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.967176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.967176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.967288 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.967358 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.967627 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.967710 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 13:42:09.967971 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.959192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.126:37011 (local address 127.0.0.1:37310)
0924 13:42:09.959301 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.959337 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:09.959646 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:09.959822 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:09.959824 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:09.960207 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:09.960444 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.960449 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.961355 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.961356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.962387 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.962390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.962777 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.962778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.962873 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.963338 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:09.963348 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:09.964893 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:09.966500 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.966507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.966509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.966766 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.966984 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.966985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.966986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.967102 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.967360 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:09.967367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:09.967498 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 13:42:09.967606 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 13:42:09.967747 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":57,"threads_started":1}
I20260924 13:42:09.968582 16883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:09.968640 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.958854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.25.240.124:37839)
0924 13:42:09.959040 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.959051 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.959086 (+ 35us) server_negotiation.cc:407] Connection header received
0924 13:42:09.959631 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.959639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.959659 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.959772 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.960544 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.961686 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.962649 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.963032 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.963100 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.965597 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.965602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.965603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.965608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.967267 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.967660 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.967663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.967663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.967719 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.968029 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.968030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.968030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.968194 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.968361 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.968433 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.968505 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.126@KRBTEST.COM
I20260924 13:42:09.975463 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.968826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.125:38055 (local address 127.0.0.1:37098)
0924 13:42:09.968979 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.968994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:09.969035 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:09.969205 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:09.969206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:09.969353 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:09.969391 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.969393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.970110 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.970111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.970592 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.970593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.971001 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.971002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.971066 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.971455 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:09.971460 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:09.972673 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:09.974480 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.974481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.974481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.974650 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.974840 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.974841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.974841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.974902 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.975162 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:09.975162 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:09.975194 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 13:42:09.975272 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 13:42:09.975361 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260924 13:42:09.975474 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.968903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.25.240.125:38055)
0924 13:42:09.968952 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.968954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.969032 (+ 78us) server_negotiation.cc:407] Connection header received
0924 13:42:09.969127 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.969128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.969140 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.969202 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.969520 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.970015 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.970690 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.970907 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.971001 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.972759 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.972760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.972761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.972766 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.974378 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.974719 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.974720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.974721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.974773 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.974963 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.974964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.974964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.975097 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.975167 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.975317 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.975389 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 13:42:09.975834 16883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } has no permanent_uuid. Determining permanent_uuid...
I20260924 13:42:09.976631 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.969449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.25.240.126:37011)
0924 13:42:09.969489 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.969490 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.969497 (+ 7us) server_negotiation.cc:407] Connection header received
0924 13:42:09.969518 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.969519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.969527 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.969614 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.970061 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.970497 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.971156 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.971353 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.971434 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.973617 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.973618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.973618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.973624 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.975192 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.975499 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.975500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.975500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.975903 (+ 403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.976116 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.976117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.976117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.976224 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.976402 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.976437 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.976506 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.124@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.240.125@KRBTEST.COM
I20260924 13:42:09.982110 16733 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } attempt: 2
I20260924 13:42:09.982144 16887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.976097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37658)
0924 13:42:09.976176 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.976185 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:09.976247 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:09.976365 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:09.976366 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:09.976501 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:09.976522 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.976523 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.977229 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.977230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.977868 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.977870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.978203 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.978203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.978253 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.978618 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:09.978622 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:09.979783 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:09.981191 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.981192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.981193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.981370 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.981580 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.981580 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.981581 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.981635 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.981856 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:09.981856 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:09.981890 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 13:42:09.981953 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 13:42:09.982045 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 13:42:09.982313 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.976128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.25.240.124:37839)
0924 13:42:09.976159 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.976161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.976236 (+ 75us) server_negotiation.cc:407] Connection header received
0924 13:42:09.976301 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.976302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.976310 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.976363 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.976593 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.977168 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.977949 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.978137 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.978187 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.979872 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.979873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.979873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.979879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.981097 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.981447 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.981448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.981448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.981495 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.981695 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.981696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.981696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.981792 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.981863 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.981985 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.982093 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":16}
I20260924 13:42:09.984498 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb: Bootstrap starting.
I20260924 13:42:09.984642 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.977132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.25.240.125:38055)
0924 13:42:09.977165 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.977166 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.977193 (+ 27us) server_negotiation.cc:407] Connection header received
0924 13:42:09.977266 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.977267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.977275 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.977330 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.977810 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.978376 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.979152 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.979333 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.979386 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.981684 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.981685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.981685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.981692 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.983468 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.983835 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.983836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.983836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.983894 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.984112 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.984113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.984113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.984233 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.984387 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.984460 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.984529 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 13:42:09.985630 16883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:09.985914 16883 log.cc:824] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:09.986770 26561 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 13:42:09.986811 16883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb: No bootstrap required, opened a new log
Sep 24 13:42:09 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.126@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
I20260924 13:42:09.987607 16707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37326
I20260924 13:42:09.989073 16883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } }
I20260924 13:42:09.989188 16883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c60959642142bea376fc9279610acb, State: Initialized, Role: FOLLOWER
I20260924 13:42:09.989290 16883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } }
I20260924 13:42:09.989661 16883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:09.989642 16889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:09.989727 16889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:09.990132 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.982800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.25.240.124:37839)
0924 13:42:09.982848 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.982850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:09.982859 (+ 9us) server_negotiation.cc:407] Connection header received
0924 13:42:09.982917 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:09.982917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:09.982924 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:09.983133 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:09.983288 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.983711 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.984834 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.985245 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.985307 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.987369 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:09.987371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:09.987371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:09.987377 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:09.988899 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.989301 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.989303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.989303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.989356 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:09.989579 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:09.989580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:09.989580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:09.989728 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:09.989796 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:09.989985 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 13:42:09.990052 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 13:42:09.990137 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.982430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37662)
0924 13:42:09.982801 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:09.982811 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:09.982853 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:09.982999 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:09.983000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:09.983161 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:09.983193 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.983194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.983802 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.983803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.984557 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:09.984559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:09.985305 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:09.985306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:09.985360 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:09.985762 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:09.985768 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:09.987273 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:09.988993 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.988994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.988994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.989215 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.989418 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:09.989419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:09.989419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:09.989488 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:09.989823 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:09.989824 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:09.989871 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 13:42:09.989948 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 13:42:09.990049 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351}
W20260924 13:42:09.991530 16900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 13:42:09.991518 16733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a: Bootstrap starting.
W20260924 13:42:09.991586 16900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:09.992123 16733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:09.992393 16733 log.cc:824] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:09.993211 16733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a: No bootstrap required, opened a new log
I20260924 13:42:09.994664 16733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } }
I20260924 13:42:09.994792 16733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c4dcac1f254c75bcd4a6e3c0ce0c5a, State: Initialized, Role: FOLLOWER
I20260924 13:42:09.994863 16733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } }
I20260924 13:42:09.995170 16733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:09.992877 16815 master_runner.cc:186] Error getting master registration for 127.25.240.126:37011: OK, instance_id { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" instance_seqno: 1790257329639378 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:42:09.995932 16707 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 13:42:09.995994 16707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:09.995996 16815 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:42:09.996734 16707 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } }
I20260924 13:42:09.996995 16707 leader_election.cc:288] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [CANDIDATE]: Term 1 election: Requested vote from peers af141c61560746899b968e2b6bac650e (127.25.240.125:38055), 99c60959642142bea376fc9279610acb (127.25.240.124:37839)
I20260924 13:42:09.997238 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:09.997419 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: This master's current role is: FOLLOWER
W20260924 13:42:09.997459 16914 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:09.997524 16914 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:09.997476 16857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99c60959642142bea376fc9279610acb"
I20260924 13:42:09.997622 16857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:09.998238 16857 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c4dcac1f254c75bcd4a6e3c0ce0c5a in term 1.
I20260924 13:42:09.998476 16683 leader_election.cc:302] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c4dcac1f254c75bcd4a6e3c0ce0c5a, 99c60959642142bea376fc9279610acb; no voters:
I20260924 13:42:09.998613 16903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 13:42:09.998773 16903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [term 1 LEADER]: Becoming Leader. State: Replica: 12c4dcac1f254c75bcd4a6e3c0ce0c5a, State: Running, Role: LEADER
I20260924 13:42:09.998853 16903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } }
I20260924 13:42:09.999289 16916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c4dcac1f254c75bcd4a6e3c0ce0c5a. Latest consensus state: current_term: 1 leader_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:09.999583 16916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:10.000368 16665 master_runner.cc:186] Error getting master registration for 127.25.240.125:38055: OK, instance_id { permanent_uuid: "af141c61560746899b968e2b6bac650e" instance_seqno: 1790257329796104 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 13:42:10.002595 16665 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 13:42:10.013633 16923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 13:42:10.013867 16923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 13:42:10.015110 16736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:10.005818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.25.240.126:37011)
0924 13:42:10.005860 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.005862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.006291 (+ 429us) server_negotiation.cc:407] Connection header received
0924 13:42:10.006385 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.006386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.006395 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.006425 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.007152 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.007893 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.008933 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.009236 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.009276 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:10.011236 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:10.011237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:10.011238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:10.011245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:10.013416 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:10.013961 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:10.013962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:10.013962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:10.014015 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:10.014479 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:10.014480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:10.014480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:10.014596 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:10.014670 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:10.014925 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 13:42:10.015011 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:10.015177 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:09.999360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.25.240.124:37839)
0924 13:42:09.999439 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:09.999441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.006045 (+ 6604us) server_negotiation.cc:407] Connection header received
0924 13:42:10.006221 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.006235 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.006248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.006282 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.007408 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.008096 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.009459 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.009780 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.009821 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:10.011924 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:10.011926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:10.011926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:10.011933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:10.013531 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:10.014209 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:10.014210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:10.014210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:10.014260 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:10.014460 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:10.014461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:10.014461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:10.014596 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:10.014678 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:10.015003 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 13:42:10.015082 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 13:42:10.018864 16857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [term 1 FOLLOWER]: Refusing update from remote peer 12c4dcac1f254c75bcd4a6e3c0ce0c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:10.019446 16916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:10.020748 16889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c4dcac1f254c75bcd4a6e3c0ce0c5a. Latest consensus state: current_term: 1 leader_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:10.020833 16889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:10.021840 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:10.021935 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:10.022066 16927 mvcc.cc:205] Tried to move back new op lower bound from 7332894023753068544 to 7332894023675949056. Current Snapshot: MvccSnapshot[applied={T|T < 7332894023753068544}]
I20260924 13:42:10.022121 16889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:10.022181 16889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:10.027321 16923 catalog_manager.cc:1347] Generated new cluster ID: 62a941e9e2674d85ab1def38a9b3da32
I20260924 13:42:10.027372 16923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 13:42:10.031023 16808 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } attempt: 3
I20260924 13:42:10.035109 16812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:10.006050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.25.240.125:38055)
0924 13:42:10.006089 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.006091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.014721 (+ 8630us) server_negotiation.cc:407] Connection header received
0924 13:42:10.014822 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.014824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.014835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.014864 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.015929 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.016668 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.017683 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.017940 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.018008 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:10.026592 (+ 8584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:10.026593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:10.026593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:10.026601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:10.028264 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:10.033941 (+ 5677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:10.033942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:10.033942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:10.034000 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:10.034245 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:10.034246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:10.034246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:10.034386 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:10.034485 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:10.034945 (+ 460us) server_negotiation.cc:299] Negotiation successful
0924 13:42:10.035022 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 13:42:10 dist-test-slave-m5z1 krb5kdc[16652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790257329, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.240.125@KRBTEST.COM for kudu/127.25.240.124@KRBTEST.COM
I20260924 13:42:10.040321 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:10.031243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.240.124:37839 (local address 127.0.0.1:37678)
0924 13:42:10.031286 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 13:42:10.031296 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 13:42:10.031328 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 13:42:10.031551 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 13:42:10.031551 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 13:42:10.031702 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 13:42:10.031742 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:10.031743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.032581 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.032582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:10.033355 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 13:42:10.033357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.033858 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.033859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 13:42:10.033923 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:10.034323 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 13:42:10.034328 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 13:42:10.036274 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 13:42:10.039186 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:10.039187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:10.039187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:10.039406 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:10.039641 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 13:42:10.039641 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 13:42:10.039642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 13:42:10.039709 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 13:42:10.040010 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 13:42:10.040011 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 13:42:10.040072 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 13:42:10.040136 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 13:42:10.040226 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 13:42:10.040683 16886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:10.031396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.25.240.124:37839)
0924 13:42:10.031432 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.031434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.031440 (+ 6us) server_negotiation.cc:407] Connection header received
0924 13:42:10.031459 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.031459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.031468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.031514 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.031822 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.032520 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.033440 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:10.033744 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:10.033807 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:10.036376 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 13:42:10.036377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 13:42:10.036378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 13:42:10.036384 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 13:42:10.039087 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:10.039493 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:10.039493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:10.039494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:10.039562 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 13:42:10.039781 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 13:42:10.039782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 13:42:10.039782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 13:42:10.039916 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 13:42:10.039973 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:10.040526 (+ 553us) server_negotiation.cc:299] Negotiation successful
0924 13:42:10.040600 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 13:42:10.042148 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e: Bootstrap starting.
I20260924 13:42:10.042855 16808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e: Neither blocks nor log segments found. Creating new log.
I20260924 13:42:10.043164 16808 log.cc:824] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e: Log is configured to *not* fsync() on all Append() calls
I20260924 13:42:10.053926 16808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e: No bootstrap required, opened a new log
I20260924 13:42:10.055946 16808 raft_consensus.cc:348] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } }
I20260924 13:42:10.056087 16808 raft_consensus.cc:729] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af141c61560746899b968e2b6bac650e, State: Initialized, Role: FOLLOWER
I20260924 13:42:10.056170 16808 consensus_queue.cc:260] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } }
I20260924 13:42:10.056519 16782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af141c61560746899b968e2b6bac650e"
I20260924 13:42:10.056581 16781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [term 0 FOLLOWER]: Advancing to term 1
I20260924 13:42:10.057456 16781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [term 1 FOLLOWER]: Refusing update from remote peer 12c4dcac1f254c75bcd4a6e3c0ce0c5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 13:42:10.057883 16903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 13:42:10.058312 16782 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c4dcac1f254c75bcd4a6e3c0ce0c5a in term 1.
I20260924 13:42:10.058713 16923 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 13:42:10.059459 16923 catalog_manager.cc:1504] Loading token signing keys...
I20260924 13:42:10.056507 16808 sys_catalog.cc:564] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 13:42:10.058374 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:10.062031 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:10.062176 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12c4dcac1f254c75bcd4a6e3c0ce0c5a. Latest consensus state: current_term: 1 leader_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:10.062235 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:10.062319 16932 sys_catalog.cc:455] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:10.062369 16932 sys_catalog.cc:458] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 13:42:10.063956 16934 mvcc.cc:205] Tried to move back new op lower bound from 7332894023898447872 to 7332894023675949056. Current Snapshot: MvccSnapshot[applied={T|T < 7332894023753068544}]
W20260924 13:42:10.066090 16947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 13:42:10.066146 16947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 13:42:10.072831 16903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12c4dcac1f254c75bcd4a6e3c0ce0c5a" member_type: VOTER last_known_addr { host: "127.25.240.126" port: 37011 } } peers { permanent_uuid: "af141c61560746899b968e2b6bac650e" member_type: VOTER last_known_addr { host: "127.25.240.125" port: 38055 } } peers { permanent_uuid: "99c60959642142bea376fc9279610acb" member_type: VOTER last_known_addr { host: "127.25.240.124" port: 37839 } } }
I20260924 13:42:10.072937 16903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a [sys.catalog]: This master's current role is: LEADER
I20260924 13:42:10.074433 16923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12c4dcac1f254c75bcd4a6e3c0ce0c5a: Generated new TSK 0
I20260924 13:42:10.074611 16923 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 13:42:10.096274 16886 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.094642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.25.240.124:37839)
0924 13:42:10.094688 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.094690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.095594 (+ 904us) server_negotiation.cc:407] Connection header received
0924 13:42:10.095703 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.095704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.095716 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.095747 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.096203 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37682: BlockingRecv error: recv got EOF from 127.0.0.1:37682 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 13:42:10.097021 16736 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.095081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.25.240.126:37011)
0924 13:42:10.095121 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.095123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.096515 (+ 1392us) server_negotiation.cc:407] Connection header received
0924 13:42:10.096616 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.096617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.096628 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.096658 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.096962 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37350: BlockingRecv error: recv got EOF from 127.0.0.1:37350 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 13:42:10.097790 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.095394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.25.240.125:38055)
0924 13:42:10.095453 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.095455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.097212 (+ 1757us) server_negotiation.cc:407] Connection header received
0924 13:42:10.097337 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.097338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.097348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.097379 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.097733 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37130: BlockingRecv error: recv got EOF from 127.0.0.1:37130 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 13:42:10.099980 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.099235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.25.240.125:38055)
0924 13:42:10.099265 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.099267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.099508 (+ 241us) server_negotiation.cc:407] Connection header received
0924 13:42:10.099607 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.099608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.099624 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.099651 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.099938 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37144: BlockingRecv error: recv got EOF from 127.0.0.1:37144 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 13:42:10.102825 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.102118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.25.240.125:38055)
0924 13:42:10.102148 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.102150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.102343 (+ 193us) server_negotiation.cc:407] Connection header received
0924 13:42:10.102449 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.102450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.102460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.102487 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.102786 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37148: BlockingRecv error: recv got EOF from 127.0.0.1:37148 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 13:42:10.106643 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.105908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.25.240.125:38055)
0924 13:42:10.105950 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.105952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.106158 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:42:10.106252 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.106253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.106262 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.106289 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.106604 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37158: BlockingRecv error: recv got EOF from 127.0.0.1:37158 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 13:42:10.112494 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.111753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.25.240.125:38055)
0924 13:42:10.111786 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.111787 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.111993 (+ 206us) server_negotiation.cc:407] Connection header received
0924 13:42:10.112088 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.112089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.112098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.112126 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.112454 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37174: BlockingRecv error: recv got EOF from 127.0.0.1:37174 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 13:42:10.122552 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.121736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37184 (local address 127.25.240.125:38055)
0924 13:42:10.121777 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.121779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.122012 (+ 233us) server_negotiation.cc:407] Connection header received
0924 13:42:10.122111 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.122112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.122122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.122150 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.122501 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37184: BlockingRecv error: recv got EOF from 127.0.0.1:37184 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 13:42:10.140832 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.139939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.25.240.125:38055)
0924 13:42:10.139983 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.139985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.140240 (+ 255us) server_negotiation.cc:407] Connection header received
0924 13:42:10.140347 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.140348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.140359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.140389 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.140778 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37192: BlockingRecv error: recv got EOF from 127.0.0.1:37192 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 13:42:10.175019 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.174197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37198 (local address 127.25.240.125:38055)
0924 13:42:10.174244 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.174246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.174397 (+ 151us) server_negotiation.cc:407] Connection header received
0924 13:42:10.174483 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.174484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.174497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.174569 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.174954 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37198: BlockingRecv error: recv got EOF from 127.0.0.1:37198 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 13:42:10.240794 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.240101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.25.240.125:38055)
0924 13:42:10.240153 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.240155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.240266 (+ 111us) server_negotiation.cc:407] Connection header received
0924 13:42:10.240300 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.240301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.240313 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.240391 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.240732 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37208: BlockingRecv error: recv got EOF from 127.0.0.1:37208 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 13:42:10.370762 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.370036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.25.240.125:38055)
0924 13:42:10.370106 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.370108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.370234 (+ 126us) server_negotiation.cc:407] Connection header received
0924 13:42:10.370280 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.370281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.370292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.370355 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.370679 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37218: BlockingRecv error: recv got EOF from 127.0.0.1:37218 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 13:42:10.628733 16812 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 13:42:10.627959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.25.240.125:38055)
0924 13:42:10.628047 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:10.628050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:10.628131 (+ 81us) server_negotiation.cc:407] Connection header received
0924 13:42:10.628205 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:10.628206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:10.628217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:10.628288 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 13:42:10.628671 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37228: BlockingRecv error: recv got EOF from 127.0.0.1:37228 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 13:42:10.992008 16900 catalog_manager.cc:1259] Loaded cluster ID: 62a941e9e2674d85ab1def38a9b3da32
I20260924 13:42:10.992084 16900 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb: loading cluster ID for follower catalog manager: success
I20260924 13:42:10.992900 16900 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb: acquiring CA information for follower catalog manager: success
I20260924 13:42:10.993229 16900 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99c60959642142bea376fc9279610acb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:42:11.066661 16947 catalog_manager.cc:1259] Loaded cluster ID: 62a941e9e2674d85ab1def38a9b3da32
I20260924 13:42:11.066743 16947 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e: loading cluster ID for follower catalog manager: success
I20260924 13:42:11.067503 16947 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e: acquiring CA information for follower catalog manager: success
I20260924 13:42:11.067754 16947 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af141c61560746899b968e2b6bac650e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 13:42:11.144670 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:11.142032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.25.240.125:38055)
0924 13:42:11.142142 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:11.142144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:11.142158 (+ 14us) server_negotiation.cc:407] Connection header received
0924 13:42:11.142236 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:11.142237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:11.142304 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:11.142393 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:11.142566 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.143059 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.143853 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.144060 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.144140 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:11.144307 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:11.144418 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:11.144514 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:11.144548 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 13:42:11.144570 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":50,"threads_started":1}
I20260924 13:42:11.149032 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:11.146560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.25.240.124:37839)
0924 13:42:11.146692 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:11.146694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:11.146702 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:11.146738 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:11.146738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:11.146774 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:11.146845 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:11.146985 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.147462 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.148208 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.148424 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.148508 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:11.148685 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:11.148798 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:11.148887 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:11.148910 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 13:42:11.148933 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260924 13:42:11.153379 17023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:11.150687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.25.240.124:37839)
0924 13:42:11.150723 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:11.150725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:11.150848 (+ 123us) server_negotiation.cc:407] Connection header received
0924 13:42:11.150912 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:11.150913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:11.150926 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:11.150982 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:11.151106 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.151831 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.152632 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.152867 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.152923 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:11.153096 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:11.153131 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:11.153182 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:11.153285 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 13:42:11.153308 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 13:42:11.153862 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:11.150843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.25.240.125:38055)
0924 13:42:11.150883 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:11.150884 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:11.150988 (+ 104us) server_negotiation.cc:407] Connection header received
0924 13:42:11.151067 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:11.151067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:11.151077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:11.151128 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:11.151324 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.152008 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.152984 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.153271 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.153334 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:11.153475 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:11.153512 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:11.153659 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:11.153745 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 13:42:11.153768 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 13:42:11.154187 17031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 13:42:11.150843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.25.240.126:37011)
0924 13:42:11.150957 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 13:42:11.150959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 13:42:11.150967 (+ 8us) server_negotiation.cc:407] Connection header received
0924 13:42:11.151009 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 13:42:11.151010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 13:42:11.151091 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 13:42:11.151173 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 13:42:11.151293 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.151791 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.152852 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 13:42:11.153181 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 13:42:11.153249 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 13:42:11.153575 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 13:42:11.153710 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 13:42:11.153766 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 13:42:11.154074 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 13:42:11.154100 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20260924 13:42:11.160359 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 16665
I20260924 13:42:11.166018 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 16740
I20260924 13:42:11.170746 26561 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFHYdCJ/build/release/bin/kudu with pid 16815
2026-09-24T13:42:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2820 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5637 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (268322 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST